I asked a similar question on diy site and couldn't get a clear answer explaining how high voltage is too high..
Anyway about the electrical phenomenon: It's just the simple Ohm's law. You have wires which have some resistance and you have current going through them. Usually there should be no current going through the ground wire, so voltage drop across it is zero and you get zero volts. On the other hand we have current going through the neutral wire and it's acting as a resistor, since it has some low resistance. You're here simply measuring the voltage drop across it.
There's the second part of the story too: The neutral wire should be ground-referenced somewhere but it can happen that the reference of the ground at that location is different from the ground reference at the location of the building's ground connection. This can happen for example in TT type earthing.
Similar effect could appear in TN-C-S type earthing where the neutral and ground are connected together at some point. Since there's no current going through the ground wire and there is current going through the neutral wire, the neutral wire will again look like a resistor up until the point where they join together.
Also I forgot to mention two more reasons which may make a difference: The power system is AC and that leaves it open to inductive and capacitive coupling. Since AC can go through a capacitor, it can go through two wires which are next to each-other. The insulation sizes are such that the effect can be very weak, but in some cases it can produce measurable voltage. Same thing goes for inductive coupling: Even a straight wire has inductance and two wires running next to each-other will have mutual inductance. At mains power frequencies, the effect should be very weak, but it could contribute to the voltage.