I'm not entirely sure what your end goal is. Are you running your ground station on LiPo's? My first approach would be to run a ground station on 12V lead-acid, aka car battery, and leave the LiPo's for the flying thing.
Anyway, here's my attempt at a solution:

simulate this circuit – Schematic created using CircuitLab
It's a classic S-R latch between the two NAND's, that provides a definite on-off signal to both battery connectors. You could use NOR's instead if you happen to have those; search google images to see how.
The latch provides a fast switch time with no "bouncing" like a raw switch does. Depending on how the circuit actually works with your parts, you may actually need to slow down the one coming on to avoid connecting both batteries in parallel.
The connector itself needs a bit of explaining. The reverse-series connection of two FETs accounts for the parasitic diodes that you can't get away from. They're wired to turn on and off together, but the common reference for a control signal is not entirely predictable as to what voltage it is, hence the isolators O1,2. When the LED is on, the transistor conducts, and when it's off, it doesn't. The selective shorting of the zener diode turns the FETs off, or allows them to float on via R5,6 and a parasitic diode. The 5.1V zener diodes ensure that the gate never exceeds the rated voltage as specified in the FETs' datasheet. Feel free to change the zener to match your specific FETs.
When you first plug in a battery, its FETs float on and provide power immediately to the rest of the circuit. At this point, it's not guaranteed to select that one on its own unless you hold its select button. Or you could allow it to randomly choose one and reset itself until it chooses right.
Select 1,2 could be replaced with a toggle switch like you mentioned originally. There's no harm in holding a selection like that, as long as it's break-before-make. It gets really confused if you try to select both at the same time.