Decoupling Caps are used to avoid the noise/glitch on the power supply line. Basically when you say noise, it can be of many types. For the decoupling caps, one of the primary advantage is to remove the ground bounce(from Ground plane) & voltage sag (from voltage rail).
Inside an IC, using NMOS & PMOS circuit, the switching happens. In the figure, when the signal is pulled to VSS, there will a voltage drop across parasitic inductor as shown which will lead to sagging of voltage & when the signal is pulled down to ground, there will be a voltage drop on the ground side parasitic inductance which leads to ground bounce.
As a board designer, you can not do anything about it. So, to remove this issue on the board level, which is caused by parasitic inductance of trace & plane, we add a decoupling capacitor to provide a local path of voltage & ground. During fast switching , the capacitor acts as a decoupling element to reduce the drop across parasitic inductance. The board level figure with decoupling capacitor is given below :-
The farther the capcitor is , the more is the trace length & the more is parasictic inductance. So, it is advised to place it as close to the voltage or ground pin as possible.
It is a trade off or vendor recommendation to put it near to voltage pin or ground pin.