(1) The charger will stop charging when the battery is fully charged". Overcharging will never occur due to the charger IC being permanently applied.
(2) If you apply the charger to a fully charged cell the worst that is liable to happen is that V_max_charge (typically 4.2V) will be applied for a brief period, the battery will start to accept a small amount of current < I_charged_minimum and charging will then stop again. How long this small charging "blip" lasts for is up to the IC designer but it will probably be a small fraction of a second. This will have the effect of very slowly adding to the battery charge even when it is deemed fully charged, but the rate will be very low. eg if you cycle every 60 seconds so that a charging "blip" occurs every 60 seconds and if the blip lasts 1 second (probably far less) and if I_charging_min is 25% then the effective charge rate is Imax x 1s/60s x 25% or C/240 for a typical cell rated at max charge rate of C. It will probably be less to much less than this in practice.
(3) I note below the use of relay or MOSFET as a switch. If a MOSFET is used in eg a simple high side switch arrangement then there is the potential for each to back discharge into the charger circuit when the MOSFET is off if Vbat is more than the voltage drop of the MOSFET's protection diode. This complicates the power switching circuitry but not the control logic. If MOSFETs are used it may be necessary to use back to back MOSFETs in each channel.
(4) The TP4056 charger is low cost compares to the battery cost and probably overall cost. (About 30c or less in 1000 quantity, about $1 to $1.50 each in 10 quantity, free postage from China). It is probably easier to use 1 x TP4056 per battery. These can be enabled successively by using the /CHG output of the first to enable the second. A string of these could be joined in this way with charging rights rippling down the line. ICs that have fully charged their cell will hold it in the charged state and if a cell drops below Vmin_charged the IC will raise its /CHG line and disable any downstream chargers for as long as is required to top up the cell. The internal MOSFET switch in the TP4056 makes this a very economical approach in component count and spcace as well as in overall cost.
I have devised an arrangement that meets your specified "charge A, charge B, turn off spec" using 1 x hex Schmitt inverter package, 3 R, 1 diode, 2 C and a pushbutton switch. It provides non overlap FET switching. I can post the circuit later if of interest.
If you do not use a temperature sensing NTC per cell then you can do what you want very easily using two single pole single throw switches (which could be two suitably related MOSFETs or two relays) plus a means of toggling between them with a period of dead time. The "dead time period could be very short indeed - any time longer than somewhat longer than the time taken for the MOSFETs to switch.
If you wanted a very simple method you could use RC delay on gates with diode off and resistor on so off time constant is short and on is long. Adding a Schmitt gate in that process applies non overlap on off signals at FET gates.
You suggested charging first one cell then the other - but this is not essential. You may wish to do this to ensure that one call is charged asap but the IC will charge the battery appropriately for its current state (CC, CV, finished) without "keeping track" of current battery state.
The easiest method is to alternate between the cells at say 1 minute intervals. You could 'swap back sooner' if the /CHRG pin goes low during this period. This speeds up the charging but is not essential. Just: Turn on MOSFET-! for 1 minute, turn off MOSFET_1, turn on MOSFET_2 for 1 minute, turn off MOSFET_2, repeat ...
I am not specifically familiar with the 4056 but it probably checks for Vbattery > Vmax when power is applied. If Vbat > Vmax no action is taken. If Vmin < Vbat Vmax it will apply CC (constant current) until Vbat > Vmax then swap to CV mode. If the battery is removed Vout can be designed to either go high (add a capacitor) or go low (add a resistor). Which you do will affect what it does BUT when you connect the other battery it will promptly drop back into the charging algorithm. Worst case it will attempt to apply CC when the battery is actually charged the maximum voltage applied will be Vmax, the battery will draw < I terminate and no charging will occur. You can then wither leave it to sit for the rest of the cycle or use the "/charged" signal to toggle the clock.
More later IF WANTED ...