This is an extremely nice experiment!
I have an idea to add to jonk's comment:
It is super-good that you are testing out what happens like this! As already provided, the answer is that the battery is under some strain when powering the lights and it's voltage "droops" a little under that load. When a battery gets nearer its end-of-life, it will droop even more.
Put the batteries in the fridge for an hour or so, until they're cold (you can freeze them, too, but don't go lower than -20 °C), and repeat your experiment. You'll see that they droop a lot harder now! Turn off and wait until they have room temperature again: They should start working just like before.
What's happening?
In a battery, a chemical reaction leads to the two battery contacts having different electrical potentials – there's a voltage between them!
When you connect two things, your battery contacts, that are at different voltages, current starts flowing. That's what lights up your light bulb!
Now, imagine there's a small resistor (like, 2Ω) between your battery and the positive probe of your multimeter. It's not actually there, but you "feel" its existence:
A resistor shows a voltage drop when a current flows through it. In your case, a couple hundred milliampere flow through the bulb, the switch and back into your battery – and that leads to a couple hundred millivolts in voltage drop over that "imagined" resistor.
That's what we call "internal resistance". It's the imperfection of a voltage source (like your batteries) that leads to lower voltages the more current you draw.
Internal resistance can be many things – first of all, real batteries are made of real materials and real materials have resistance. But for batteries, that's usually just a small part of the internal resistance. The larger part is that to make a current flow, the chemical reactions (and ion wandering) inside must happen fast enough. If there's more current draw than the chemical reaction can sustain, the voltage drops.
Now, when you cooled down your batteries, you slowed down all the chemical reactions inside, and especially how fast charged atoms can wander inside the battery. That's the reason why we have fridges and freezers: With all chemical reactions being slowed by lower temperature, foodstuff doesn't go bad as quickly, because all the things that make food go bad (that is, bacterial growth, and chemical decomposition of things) simply happen in slow motion.
With the chemical reaction in the battery slowed down, the battery simply can't "keep up" as good with the current draw, and the voltage drops even further than with a warm battery.