I think people are distracted by the PMOS, but the real reason is a voltage spike when the 24V is connected, due to inductance of the wiring that is even higher when the multimeter is connected, vastly exceeding the max voltage of the LDO. I assume you are using a ceramic capacitor at the LDO input, which is a reason someone may be experiencing this problem for the first time -- it wouldn't happen with an electrolytic cap, as I'll explain.
Forget the transistor for now and model the circuit like this, with a source that has some resistance and inductance in the wiring (R1, L1). Also model the capacitor as having a very low ESR (R2):
simulate this circuit – Schematic created using CircuitLab
Simulate this and you'll see that the RLC circuit rings, peaking at about 42V when the 24V turns on:
Now try the same thing after changing R2 from 10 mΩ (which could be a typical ESR for a multilayer ceramic capacitor) to 1 Ω (which could be a typical ESR for an electrolytic capacitor) and the peak will be much lower, about 27V:
The added resistance in series with the cap significantly dampens the oscillation, keeping the input voltage well with range of the LDO. Yay! So the simple solution is to add an additional 1 Ω resistor between your 1 uF capacitor and GND to account for the low ESR of the cap.
You should be able to observe this behavior in the real circuit pretty easily with an oscilloscope. Remove the LDO (to avoid blowing it up) and watch the voltage across the capacitor when you first attach the 24V. You can tweak the value of the added resistor as necessary to ensure the peak never goes above the LDO's limit.
Alternately, the ringing will never exceed twice the input voltage regardless of any of the parasitic values, so you could switch to a LDO that has a >48V input limit and you'd always be safe.