Ground loops are important and safety is always a major consideration BUT from a technical point of view, use of the ground AT the probe tip is essential in many circuits.
The oscilloscope "sees" a voltage between probe tip and some reference point.
If the reference point is the probe-ground the ground lead can be connected very close to the point under measurement at a point where "ground" is meaningful for the signal concerned.
If the circuit is operating at say 100 MHz and involves other than perfect infinite ground planes then local ground for a signal may be quite different in potential than the "ground" level on the oscilloscope itself. Failure to use probe ground will often produce a signal that would be ideal for use as a source of random numbers.
So, for technical reasons, in complex situations, use of a probe ground connected AT the point of relevance is utterly essential. Safety and ground loop and other considerations are not made any less important by this need, but must be worked out AFTER the technical ground point requirement is met.
Achieving anything worthwhile was never meant to be easy :-)
How real men (and women) do it - a picture from a Jim Williams application note: