This will not work in the general case.
The charger board, however it looks like, is in fact a short between the two of its terminals (usually the negative ones) and a variable resistance between the others (the positive ones).
So your drawing depicts a dead short circuit.
Cells (or any other electronic elements for that matter) cannot be connected in series and in parallel at the same time. You have to actively switch between the two states. A single DPDT switch with an adequate current rating will do. As a bonus, you will need a single charging board.
In fact, this is done in a lot of (older) designs. The newer approach usually utilizes some kind of switching-mode voltage converter.
Another viable approach is to use a galvanically-separated chargers. E.g. leave off the left side of the circuit and use a separate USB charger for each battery. Or at least two for each of the two battery banks. Messy, isn't it?
And third, you can simply get a 7.2V (2S) charger instead of 1S ones you are planning to use.