What you've observed is called "dielectric absorbtion" or "recovery voltage phenomenon".
It's cause by kind of interia of the dipoles (ions) in the electrolyte while charging and discharging.
Dielectric absorption is the name given to the effect by which a capacitor, that has been charged for a long time, discharges only incompletely when briefly discharged. Although an ideal capacitor would remain at zero volts after being discharged, real capacitors will develop a small voltage from time-delayed dipole discharging, a phenomenon that is also called dielectric relaxation, "soakage", or "battery action". For some dielectrics, such as many polymer films, the resulting voltage may be less than 1–2% of the original voltage, but it can be as much as 15% for electrolytic capacitors.
When the capacitor is discharging, the strength of the electric field is decreasing and the common orientation of the molecular dipoles is returning to an undirected state in a process of relaxation. Due to the hysteresis, at the zero point of the electric field, a material-dependent number of molecular dipoles are still polarized along the field direction without a measurable voltage appearing at the terminals of the capacitor. This is like an electrical remanence.
From a Mouser note
7 Recovery Voltage
Where a capacitor is once charged and discharged with both of the terminals short-circuiting and then left the terminals open for a while, a voltage across the capacitor spontaneously increases again. This is called “recovery voltage phenomenon”. The mechanism for this phenomenon can be interpreted as follows:
When charged with a voltage, the dielectric produces some electrical changes within, and then the inside of the dielectric is electrified with the opposite polarities (dielectric polarization). The dielectric polarization occurs in both ways of proceeding rapidly and slowly. When a charged capacitor was discharged until the voltage across the capacitor disappears, and then being left the terminals open, the slow polarization will discharge within the capacitor and appear as recovery voltage. (Fig. 28).