The output of a switching converter will always have some ripple at the switching frequency. The amplitude of this ripple depends on input voltage, output voltage, choice of components, etc. To measure this ripple you need to use the proper probing technique: either X10 probe with short ground spring, or coax soldered to the board with SMD 50R resistor. This is to avoid the ground alligator clip wire from the probe picking up stray magnetic fields from the DC-DC, which can make the measurement completely useless. It is preferable to probe at the load or on previously arranged test points. Probing on a SMD capacitor will underestimate the ripple, because the capacitor has low impedance at the ripple frequency.
If the load draws low current, most switching converters will enter power-saving mode. The details of how it works depends on the particular chip, but the end result is always the same: the chip wakes up, pumps some energy into the output by switching once or a few times, then goes back to sleep. Output voltage ripple will look like a sawtooth, as your screenshot shows. When the chip wakes up, it pumps some energy in the output cap which increases output voltage, then it goes to sleep and lets the output cap power the load. During this time output voltage ramps down slowly. With synchronous DC-DC converters it is sometimes possible to disable this mode to get lower ripple at the cost of light load efficiency.
When power-saving mode is active, output ripple is determined by the hysteresis of the comparator that controls sleep mode. So using a larger output cap will reduce ripple frequency, but not amplitude.
When power-saving mode is not active, a larger output cap, or some filtering will reduce ripple amplitude, but not frequency, since it's the switching frequency.
Now LDOs and other linear regulators:
These don't create any ripple, they reduce the input ripple and noise according to power supply rejection ratio, and add some noise and drift of their own.
If it is powered from a ripple-free source, a properly working linear regulator will always have a cleaner output, at the (sometimes enormous) cost of efficiency.
AMS1117 requires the correct type of output capacitor. If you test it without capacitor, then it is not in the "working correctly" case, so indeed you are measuring large output voltage variations.
AMS1117 is a bit of a clunker, quite slow and crummy, and there are lots of counterfeits. There are many better LDOs around.
Most parameters depend on output current and input-output voltage difference, so testing with only a voltage divider at the output will not reveal much.
You can't measure a 1 pixel high signal on the scope screen because it may be noise from the scope. You should use AC mode and adjust input sensitivity until you get a measurable signal that fills a few divisions on the screen.
The difference between a voltage reference chip (Vref) and a linear regulator chip (LDO) is that the Vref chip is optimized for stable accurate output voltage with a low, constant load whereas the LDO is optimized less for accuracy and more for powering variable current loads.
Pulling a variable current from any of these will result in output voltage variations which will most likely exceed the chip's output noise spec: a stable voltage source only exists if the current drawn from it is also constant.
