AM over modulation causes the carrier wave to invert it's phase when the modulating signal has an amplitude that is above a certain level. Broadcast AM typically never does this because the complexity of an accurate demodulator is too great for the thousands and millions of receivers.
AM is just the mathematical multiplication of two signals and regular broadcast AM remains as a 2 quadrant multiplier whereas full modulation uses all four quadrants.
100% modulation is where the modulating signal drives the carrier to zero and is theoretically the maximum modulation that can be successfully demodulator by a regular AM envelope detector.
Over modulation isn't really of any significance to FM systems (unlike AM). If the modulation signal amplitude is too great, any decent frequency modulator will limit the signal so that it can't push the bandwidth of the modulated signal too wide in the frequency spectrum. In effect, the modulating signal becomes clipped.