The question is somewhat conceptual, feel free to close if it's too off-topic.
If I take my portable (battery-powered) oscilloscope and try to measure the mains voltage, the connection polarity wouldn't matter much. It would be best if the probe tip connects to hot, and the grounding clip to neutral; but swap them, and the signal will be the same (negated, of course, but if the triggering settings aren't changed, I might not even notice it). The difference is that if the grounding clip is holding the hot wire, the whole ground plane of the 'scope will be riding on that 50/60Hz sinewave. If my understanding is correct, the scope would make a weak and very inefficient antenna, radiating at 50/60Hz, but since the frequency is so low, its workings wouldn't be affected.
The question is, what happens at higher frequencies?
Again, if my understanding is correct, at sufficiently high frequency, the scope would become an antenna and this could affect its internal workings (the signal present on the grounding clip could capacitively couple to other things inside).
In general, if it's not a scope, but just a device that connects to something else's ground, and there might be high frequencies on that ground, what can be done to mitigate the issues?