It could be motion. Probably at the laser source, but it could also be at the photodiode end, even a fan somewhere can cause sensitivity.
However, if you have no aperture, or the wrong aperture, it is also possible to get stray laser paths to the sensor as the laser crosses the metal shroud of the sensor or reflects around inside the arrangement.
If you have any other kind of optical window lasers can bounce around inside those too.
Over sensitivity of the detector circuit will also cause you grief. For best results you want the detector circuit to give you somewhere in the 80-90% swing when fully exposed, not swamped. This will give you sufficient tolerance for it to work over a variety of devices and power conditions while still giving you sufficient signal range to use appropriate hysteresis.
Often people think they need to use pinpoint lasers for position detection because they think lasers are wonderful. The truth is, unless you want to position something at a distance, to <1mm accuracy, using a laser can actually cause you more grief than using a less columnar light source.
With lasers it is important to align both ends. With a simple light source and appropriately appertured receiver, you only need to position the receiver accurately.
Lasers tend to ricochet. There are occasions when the laser can actually bounce around the object you are tying to measure and still end up on the sensor. Worse they can actually bounce around within your sensor.
If the laser and receiver are meters apart you can have thermal issues. Relative motion between them due to thermal expansion of whatever they are attached to can cause the laser to miss the target entirely. In fact keeping both ends mechanically coupled is an issue in general.
In many occasions I have found it prudent to actually defocus the laser so it arrives as a quarter sized spot at the receiver end. The aperture on the detector was accurate enough for the task at hand, but the alignment and vibration issues went away.