Battery voltage doesn't usually drop just because there's a load connected. But the measured voltage does tend to drop
Here's what you've got to know about voltage measurements
A voltmeter uses a resistor with a very high resistance. Ideally, it's infinite. The voltmeter measures the voltage across this resistor.
So when you connect a battery to your voltmeter, the internal resistance of the battery is insignificant compared to the voltmeter's resistance. So most of the voltage drop occurs across the voltmeter's resistance and not the battery's internal resistor. Hence, you measure the right voltage.
However, your microcontroller may have a resistance that's not too high. If the battery had an internal resistance of say, 1 milli ohm, and the voltmeter was using a 24000 ohm resistor, this error is expected.