For TEG projects, your module selection, input and output heat sinks will greatly matter, as well as the temperature differential you're able to safely create.
Depending on the type of battery (or maybe ultracapacitors?) you use, you likely may not need a switch to your battery, which would be an advantage because at low wattages, where your voltage converter is unable to sustain 5v, 500mA or 1A, 2A etc, you can repeatedly store power and discharge it. Bear in mind that unless you want a rather heavy heat sink, you will likely want to use a portion of the power the TEG generates to run a fan on the cool side, unless you're considering a water reservoir and utilizing the boiling point of water as a temperature limit.
Having a fan is not necessarily bad, as aside from cooling your heat sink, the air from it can be used to stoke the flame you are using to keep the hot side hot. You may wish to look up the Biolite Camp stove. I have one. It's pretty good for emergency charging, although I've done a lot of research on TEGs with the interest of building my own. I'm pretty sure it operates off of a single TEG, and not a high temperature one.
It will moderately slowly (800mA) charge your USB devices as long as you keep a raging inferno going inside of it. This leads me to believe you could build a much better phone charger from 3 or 4 40mm x 40mm Seebeck generators, but it would weigh more.
I think electrical is the right forum for this, but there are a lot of physics involved for a good Peltier cooler or Seebeck generator. Consider Pyrogel high temperature aerogel insulation where you need it(quantities are so small you may find the exorbitant price acceptable), sealing your TEG with non thermally conductive sealant helps and don't overlook things like copper heat pipes(for the price, CPU heat sinks can be a great deal and if you watch your local computer stores you can probably get them in very small form factors.)
Good thermal paste or adhesive is well worth the money, and if you're buying flat peltier/seebeck modules, consider getting a Dremel (or rotary tool if you prefer) with a diamond disc blade and making yourself a holder for it so that you can easily use it like a table saw (or over the top) and cut half the depth of the ceramic plates without shattering them.
Then cut a grid into the plates in between the lines of Bismuth(or whatever material) blocks. This will help prevent catastrophic failure if the ceramic plates on your TEG crack due to high heat differential/thermal stress. Unless you want the charge density of a lithium ion battery, you may find the simplicity of using ultracapacitors and their light weight preferable.
That being said, I'd consider using both, charging ultracaps to a respectable voltage, and charging either the phone directly or charging the battery bank, and charging off the battery bank only when the ultracap is below a certain voltage, giving you more or less continuous output from the combination of them.