Here is an expensive but simple solution:
simulate this circuitsimulate this circuit – Schematic created using CircuitLab
I'll try to tidy up this...
If you don't want to charge from mains, substitute your AC/DC charger with a DC/DC one, but isolated/floating.