The gate of a MOSFET has a very, very high DC resistance. For all intents and purposes, it basically doesn't consume any current at all if it's just sitting at some steady-state value (we're talking femto-amps or less).
Also, MOSFET gates all have 'parasitic capacitance', which is essentially a couple of little tiny capacitors (usually a few pF) that connect the gate to the drain and the source.
When you press switch S1, you let in a whole bunch of charge from the +5V rail, which turns on the MOSFET. The trick is that it also charges up the gate's parasitic capacitors. When you release S1, all that stored charge has nowhere to go. It doesn't get consumed by the MOSFET's gate (since the gate doesn't consume any current), and also doesn't have any path to get back to ground.
Since the charge has nowhere to go, it just sits there and maintains +5V on the gate until you connect something else (such as S2 or your multimeter) and provide a path for the charge the take back to ground.
edit: fun fact, this phenomenon is also exactly how NAND Flash works.