I have been wondering this for a while, so I did some digging, and as usual, wikipedia was the most illuminative. It looks like the telegraph was the first time ground was introduced to complete a circuit. It seemed to have been a good idea to save wire if not for the fact that ground is going to have a lot more resistance, so all in all you are going to need a lot more power to get the electricity to flow.
This was fine for telegraphy, but for anything else, it seems to more trouble than its worth.
I have read the argument for a "reference point" in measuring absolute voltages. That is solution to a contrived problem. Voltages are only meaningful as potential differences. Therefore, you measure potential differences, not choose a meaningless reference frame.
As for the safety argument, it seems we would be a lot safer without the grounding system. If ground was not used to complete the circuit, electricity would generally have to reason to travel through a person into the ground. This would reduce risk of electrocution by requiring someone touch both wires to get shocked rather than just the hot one. The ground wire seems to be a safety measure taken to avoid an unnecessary problem.
As for low voltage systems, the ground creates an unnecessary source of noise. As far as I can tell, the ground would act as a giant unstable battery+antenna. Any ph difference in the ground would mean a dc offset--dependent on the difference in acidity of the water and mineral compositions at the two ground points.
Using the ground to complete the circuit for radio signals means unnecessary resistance in a circuit. Your quality is at the mercy of your choice in soil rather than the design of your circuit.
There are probably problems I missed, but I am really wondering whether there are some really sensible advantages I missed.
Does anyone know the history/philosophy of using ground to complete circuits?