You could put a cap to ground at/near your mosfet gate. Perhaps across \$R_4\$ in your circuit.
Try this as an option:

simulate this circuit – Schematic created using CircuitLab
\$R_3\$ limits sudden currents to \$C_1\$ and the gate of \$M_1\$ and oscillations, but still also allows \$C_1\$ to discharge pretty fast through the LED load when the power is removed. \$R_4\$ is needed to keep trace currents from charging the mosfet gate and to also further discharge \$C_1\$ when the LED isn't conducting much anymore.
The above circuit is set up for an LED current of about \$20\:\textrm{mA}\$ as an example. If you plan some monster LED later on, the values and part selections would need to be changed.
EDIT:
I'd much have preferred the use of another BJT rather than a mosfet. In that case, the following circuit is an easy derivative, robust, and doesn't need \$C_1\$:

simulate this circuit
You could also get rid of one (or perhaps even both) of the two \$33\:\textrm{k}\Omega\$ resistors and further simplify it. The LED load is plenty, already, so I don't see much real need for \$R_4\$. And given that, the whole thing should power up in the right state also without the need for \$R_2\$. But this applies so long as there is a load. Otherwise, I'd keep at least one of them.