Floating power supply: voltage difference fixed but separate voltages not?

I have a +/- and GND power supply. The GND should be fixed to earth ground, and asking the supply for 5V gives $\Delta v = v_+ - v_- = 5V$.

If I use a voltmeter against the positive and ground terminals, I get $v_+ - v_{GND} = 0V$. No problem, the supply is floating. If I also check the difference between the ground and negative terminals, I get $v_- - v_{GND} = 0V$.

That's confusing, because $\Delta v = v_+ - v_- =(v_+ - v_{GND}) - (v_- - v_{GND}) = 0V \ne 5V$. So the + and - terminals of the supply are constantly floating and not fixed, but their difference is fixed? Otherwise if either of the terminals was fixed, I should see a difference between that and GND.

Also, is there any benefit (e.g. protection from leakage current) to short the GND and - terminal and get my voltage off the + and GND connections rather than +/-? I'm concerned about damaging my device.

If your supply is truly floating output then the act of connecting the meter will ground that terminal through the (high) internal resistance of the meter. simulate this circuit – Schematic created using CircuitLab

Figure 1. (a) Your first test. (b) Your second test. (c) The test you didn't do.

If you try a second meter simultaneously with the first - or a resistor with the same impedance as your meter input (10 MΩ is typical for DMMs) - you should get close to the readings of Figure 1c.

That's confusing, because $\Delta v = v_+ - v_- =(v_+ - v_{GND}) - (v_- - v_{GND}) = 0V \ne 5V$.

The fallacy with your maths is that the two conditions $v_+ - v_{GND}$ and $v_- - v_{GND}$ were not measured simultaneously. The act of measuring one altered the other and vice versa. If you measure both simultaneously the maths will work out. In Figure 1c this would be $\Delta v =(v_+ - v_{GND}) - (v_- - v_{GND}) = 2.5 - (-2.5) = 5V$.

So the + and - terminals of the supply are constantly floating and not fixed, but their difference is fixed?

Correct.

Also, is there any benefit (e.g. protection from leakage current) to short the GND and - terminal and get my voltage off the + and GND connections rather than +/-? I'm concerned about damaging my device.

No, but it's not recommended. The return wire should go to the V- terminal as that's where it ultimately needs to return to. The problem arises not because of any real electrical reason but because you may cause confusion when you or another has to maintain it later. In addition, you have to consider what will happen if the V- to GND link becomes disconnected.

Try using two identical meters simultaneously. Then you should see approximately 2.5 volts on each meter if both outputs have a decent impedance to ground balance.

Also, is there any benefit (e.g. protection from leakage current) to short the GND and - terminal and get my voltage off the + and GND connections rather than +/-? I'm concerned about damaging my device

None that I can see. Plenty of circuits use and need bipolar power supplies such as instrument analysis circuits. Thermocouples tend to get earthed in plenty of circumstances so a positive and negative supply with respect to earth is a must for many thermocouple amplifiers.

When you measure the voltage between either terminal of your floating (means "not connected to "ground"") power supply, the meter is really saying "I can't measure a voltage as there is no complete circuit" rather than "Zero Volts".

Unfortunately, the meter has no way to say "no connection" other than to display Zero Volts.