Why does the signal on the oscilloscope become less stable when I
reduce the voltage from the function generator
The oscilloscope does not automatically scale the input. The main problem is that you don't have a high enough signal to noise ratio on
Channel 1 for reliable triggering. It's not necessarily noise inherent to your system, just the limits of the oscilloscope settings.
Try this: set your trigger mode to Normal and capture a single sequence until you get a trigger that "looks" wrong - meaning it triggered in the wrong place. This usually happens on the opposite edge, where a bit of noise hits right at the trigger voltage.
As st2000 points out, this isn't an artifact of the "Auto" trigger mode as the oscilloscope believes it's triggering properly. You would see the same thing even if you had the scope set to "Normal". You can see for yourself the difference in trigger modes by setting the trigger voltage to something way out of the range of the input signal, such as +5V. After it doesn't trigger for a while, it will start showing signals, but they will appear to move incoherently (because the trigger isn't synchronizing the waveforms to the same point of the curve).
In general, the best volts/division setting for each channel is the lowest volts/division possible without clipping the signal. This really helps digital oscilloscopes provide the best accuracy in measurements even if it doesn't "look" as good.
- Recommended: change the input volts per division on
Channel 1 to something much lower, such as 500 mV/division.
- Also works: Trigger off channel 2 (especially when it's the "cleaner looking" signal).
- For tough (periodic) cases: Increase the trigger hold-off time to greater than half the period of your signal to prevent the oscilloscope from even considering the wrong crossing.