Searching infos to the Wemos D1, I landed here.
Sadly the links given in OP don't work anymore, so I'm a little confused. Though I know this is a 4 year old post, I want to answer.
The broken link named "Wemos D1 mini V2.0" as exactly this product name has a regulator (ME6211) behind the 5V pin that accepts at max 5.5V. It won't explode immediately if you feed it with 5,6V, but powering it with 6V is probably not the best idea.
I don't know where the above mentioned maximum 24V come from. There is another ESP called NodeMCU (that's not a D1 Mini!) Some of those boards have an SPX3819M5 regulator that can handle up to 16V.
From running multiple D1 with li-ion batteries, I can say it still works with voltages down to 2,8V on the 5V pin. The decreasing voltage while discharging is not a problem at all.
I suggest instead of using 4x1.5V use 3x1.5, so the maximum is just 4.5V and it will work even if the batteries are almost discharged (then it's 3,6V.)
Another alternative I already have in use is running 3xAA rechargable batteries, which fully loaded provide up to 4.3V. I usually recharge them when they reach 3.6-3.5V.