I have a DMX WiFi receiver powered by a 3.7V 650mA battery. Link to a similar product
When I don't connect an external power supply nothing starts (no LED, etc.) but when I start with the external power supply and then unplug it my receiver works on battery as it is supposed to (and for the specified amount of time so like 8 hours IIRC.)
My guess is that the battery doesn't deliver enough current at startup without the external power supply or maybe a capacitor on the microcontroller.
I included pictures if you guys want to see the PCB. (I disconnected the battery in the pictures.)