A voltage regulator is equivalent to a variable resistor. It follows with the voltage change of the output terminal and regulate at a fixed value. From this point, the regulator's equivalent resistor and load resistor form a voltage divider.
But there is a response time on regulator, so the output voltage will jump (swing) around the fixed value.
The capacitor is a energy storage device, and the voltage across the capacitor can not change instantaneously. So if you want to get a smooth change, a capacitor is needed.
You also need to know, the capacitor have an equivalent series resistance (ESR), and this will cause ripple. To reduce this, chose a low-ESR capacitor, or a few capacitors in parallel. (But for some circuits, a low-ESR capacitor will cause the circuit to oscillate.)
The wire also has a resistance, and it will cause ripple when you're using the capacitor. So you need place the capacitor close to the output pin of the regulator IC.