Such chargers1 are (at least, supposed to be) basically just constant-voltage supplies, i.e. whenever connected to AC power they should offer the specified voltage, no matter if a device is already connected. For a modern switching power supply, there's quite a lot of circuitry included that makes sure this goal is met pretty reliably, so indeed you don't need to worry about anything here.
But older power supplies are often built much simpler: occasionally, these would consist only of a transformer and a roughly smoothened rectifier.
simulate this circuit – Schematic created using CircuitLab
When connected to a device while not on AC, nothing much happens at all (a simple diode can make sure no current is "draining back" from the batteries), so we shouldn't worry about that.
However, the other case, connected to AC but not to any device, is not so uncritical. The thing about transformers is, they only work properly (in the sense of, providing a fixed ratio between the voltages on both sides) when there's a current in both windings. However, when running idle, the voltage in the secondary coil exceeds the nominal voltage. That's especially a concern because the smoothening capacitor will then charge up to that voltage, and if you connect a device at that instant it may suffer damage from the excess voltage.
1 Note that a properly designed actual battery charger is not a constant-voltage supply, but the part that controls this is nowadays build into the device rather than the power supply.