I think the reason seems easy to see. You observe the capacitor voltage after the ESR, which is always forming a divider network with your varying load. Let's look at the circuit for a moment (I'm sorry you didn't post it as part of your question, so I apologize for any errors I make in interpreting your words here.)
simulate this circuit – Schematic created using CircuitLab
(The above uses an ideal capacitor and adds the ESR externally for illustration.)
Now, let's run this over a 1 second period and plan on engaging the switch SW at 500 ms, for a period of 1 ms.
From your measurement point, the voltage is at first:
$$V_o = V_C\frac{R}{R+ESR}\approx .999 V_C$$
The voltage, \$V_C\$, declines for the first 500 ms until it reaches about \$6\:\textrm{V}\$, or so, and then suddenly the switch SW is engaged for 1 ms. During this moment, the divider is now:
$$V_o = V_C\frac{R\vert\vert RSW}{R\vert\vert RSW+ESR}\approx .091 V_C$$
In short, a LOT less than before. You only see a portion the voltage on the capacitor for this moment.
Then the switch SW releases and this returns the divider to where it was... But by this time a lot of charge has been removed from the capacitor, too. So it rebounds back a bit. But not entirely all the way, of course.
I've noted where the point of observation is located. I hope you can see why the voltage at this observation point will be very close to \$V_C\$, in the example above, when the switch isn't engaged. Because the current from the capacitor is so low, the \$ESR\$ doesn't drop much voltage and so the measurement point "sees" a voltage that is very close to the actual capacitor voltage at each instant.
However, when the switch becomes engaged, the current from the capacitor increases dramatically because of the suddenly reduced load. As a result of this increased current now flowing also through the \$ESR\$, it drops a much larger voltage than before, causing a larger error in the voltage measurement of \$V_C\$ as seen from the observation point shown. So it is the large drop across the \$ESR\$ that creates that sharp drop in observed voltage. But when the switch is opened again, the capacitor's current returns to a lower value again and this causes a smaller voltage drop across the capacitor and this brings your observation point once again close to the actual voltage on the capacitor.
It just makes sense to me. I hope you can see what is taking place, now.