To add a practical example...
Have you ever found an old flashlight (especially incandescent), turned it on and thought the batteries were good only to see it dim and possibly turn off several seconds later? Or possibly a motorized toy?
When the batteries aren't being asked to do any work, there may still be enough of the active chemicals to raise the voltage to nearly-new levels, but not enough to sustain that voltage under load. Once you've stopped using them, the voltage will gradually climb back up again.
An ideal multimeter in Voltage test mode draws no current, so you're not testing the battery's ability to sustain the voltage when doing work. No multimeter is ideal, but even a cheap one draws little current.
It doesn't work as well with LED's or other semiconductor-based objects because they have a relatively flat performance in the "good" voltage ranges and then just stop once they get below a certain threshold.
P.S. mA and mAh
Current (measured in Amps, Milliamps, etc.) is the measure of how fast electrons are passing through a circuit.
Capacity (measured in "Amp hours", "Milliamp hours", etc) is the measure of how long a power storage device can supply a certain current.
It is roughly true to say that a 1000mAh battery could output 1000mA for 1 hour or 1mA for 1000 hours. Reality is a bit more complicated, though. Batteries are much happier outputting 1mA than 1000mA (1 Amp). You'd have a hard time getting a 1000mA battery to output 10 Amps for 6 minutes without exploding.