FM is better at delivering higher demodulated signal to noise ratios (SNR) than AM across a large range of different carrier SNRs: -
The vertical axis is output (demodulated) SNR and the horizontal axis is carrier SNR.
As you can see, there is a term called "FM improvement factor" - this is basically how much quieter the FM demodulated output is compared to an AM scheme. Above a certain carrier SNR the FM improvement factor is constant compared to AM.
However, as carrier SNR drops below a certain level, the improvement offered by FM starts to also drop.
What is the FM threshold effect?
Usually this is defined as the carrier SNR at which the FM improvement has reduced by 1 dB. In the picture above the carrier SNR would be a little below 12 dB