A battery has some internal resistance that for Li-ion batteries is usually in the 0.15 \$\Omega\$ region when fresh.
As you should know from Ohm's Law, \$V = I \times R\$ so if you put a load on the battery this will cause a current to drain from the battery and this will cause, depending on the current, a small voltage drop over the battery's internal resistance as it would on any normal resistor in a circuit. This can be explained visually like so:
simulate this circuit – Schematic created using CircuitLab
As the battery gets used and 'ages' this internal resistance increases so the voltage drop over it will also increase for a constant current. The 'open-circuit' voltage of the battery will always be more or less the initial voltage as the 'load' will be near infinity so all the voltage drop will be over this air-gap between the batteries two terminals.
As you mentioned the drop in voltage is less with a larger load, this is because the current draw will be less and hence the drop over the internal resistance will also be less.
You should look into a constant current source using a low voltage op-amp (I did something similar to this a couple of months a go and ran into the same issue - hence that question) and you'll be able to work out the mAh easily enough as the current should be more or less constant and you'll know the time taken for the voltage to fall.
Hopefully this helps and answers your question - it is expected that a voltage drop will be seen when applying a load to a battery.