I can only speculate, as you didn't mention what you measured the input voltage at under load (or at least didn't say whether it was under load or not):
You're operating close to the lower limit of input voltage. Your batteries might be at 3.54V with no load, but the battery ESR and state of charge (as well as any wiring between it and the board) might cause it to droop when you pull current. Perhaps there exists an undervoltage mechanism to shut down the supply when the input is too low. If so, it would kick in and shut down the regulator, which would result in...that's right, lower current drain. The battery voltage quickly recovers, and at that point the regulator kicks in again.
If this is what's happening, you can measure the voltage at the input of your board and see if it's dropping to <3V when your 5V device is connected.
If that's the case, I'd suggest trying a bench supply, but I suspect this is supposed to be your bench supply. Make sure all your battery wiring is as short and as solid as possible, to eliminate that as a cause. Failing that, try paralleling more batteries to see if it makes a difference.