Here's one mental picture of a battery:
The electro-chemical reactions inside the battery happen only when there's a closed circuit. When you place a voltmeter across the poles, then you create a closed circuit (with very low current) - the electro-chemical reactions start, and you now have a voltage between the poles. The placement of the voltmeter changes the thing you are trying to measure. Without the voltmeter, there's no voltage between the poles, because there's no closed circuit.
But the above seems completely inconsistent with how people speak of open-circuit voltage, as existing independent of actually measuring it.
My reasoning must be wrong somewhere. But where?
(I know this is a very simple question, but I'm genuinely confused by it.)
Here's an example of the typical use of open-circuit voltage (see 24:00), from an MIT lecture.
Edit: replaced voltage-drop with voltage.