Like Andy aka said, the issue is the lack of common mode rejection of the measurement setup. To some extent this is to be expected when not using a differential probe, but single-ended probes work well for many practical tasks.
If the input was shorted with 0 ohm impedance, the oscilloscope would see 0 volts input. But it isn't - all of the cables in the system have significant inductance:
simulate this circuit – Schematic created using CircuitLab
The cable inductance picks up noise and causes voltage drop. In the coaxial pair, the inner and outer conductor are closely coupled and noise voltage mostly cancels out. However, the few centimeters of the ground clip wire already presents a significant unbalanced inductance.
When the noise source is floating, there is not much voltage drop over this inductance. But when the noise is coupled through mains cabling, current will flow through the probe cable outer conductor.
At the oscilloscope end, the outer terminal is connected directly to the power supply ground. Thus it presents a low impedance to any ground-referred noise source, and the noise voltage at oscilloscope ground terminal is small - most of the voltage is lost in the cable inductance.
That sounds like a good thing, but the positive terminal presents a high (typically 1 Mohm) impedance. The noise coupled there is not attenuated, and is detected by the input amplifier. Cancellation doesn't occur because the common ground causes attenuation at the negative terminal.
The source of the high frequency noise is probably some mains-powered device nearby. For me, it has usually been cheap LED lighting. Try disconnecting devices and see if affects the detected noise.
Switching to a shorter ground clip cable on the oscilloscope or to a ground spring will also give a big improvement. Here is a comparison I did with 10 MHz 20Vp-p square wave from a signal generator. White line is with 10 cm long ground clip cable, and yellow line is with ground spring (both shorted to the probe tip, like in original post):