fully charging the batteries will obviously fry your chip. don't ever charge them somewhere else.
in your setup, the batteries will never exceed 1V (*5), which is ok for your chip. 1V (*5) is the cut off voltage for NimH cells. They will always be almost empty, so a short battery life and fast capacity losses of your cells.
Can you add another cell and use converter? A linear converter would probably fit your needs, a step-down converter helps with a longer battery life
Edit: you can still use your phone charger with a switch, either manually controlled or a simple relais to switch to external supply voltage. Charging batteries is more complicated and I suggest using an external charger, you probaly own anyway.