A piezo transducer is a capacitor, and capacitors exhibit an effect called dielectric absorption, in which a capacitor which has been held charged for a long time, then discharged, may then recover some voltage across its terminals.
Dielectric absorption could account for your observations. I don't know whether it would be of sufficient magnitude to be audible.
If you're wondering why the circuit would be holding the piezo element at a nonzero voltage: for a given supply voltage, the maximum volume is obtained by not just driving one side high and low, but by driving both sides with opposite polarity (much like a motor reversing circuit), thus achieving a peak-to-peak voltage of twice the supply voltage. If the circuit driving the piezo did not have a separate "off" state (which there would be no reason to need) then it would have been held charged as long as the device was powered.
By the way, a point of terminology: a “piezo buzzer” is specifically a device which contains an oscillator circuit, so it makes a (constant) tone if you apply DC. A piezo transducer (piezo disc, piezo element) without this circuit requires an actual audio waveform — if you apply DC then you'll only get a click on applying power (like the ones you heard).
The dielectric absorption theory is more likely if there is no oscillator circuit and you have a bare transducer (since the oscillator would likely drain the tiny amount of stored energy or otherwise cause what you're seeing not to happen).