The diode does not "remove the carrier", but what it does do is "add the baseband" signal to the mix. The output of an ideal diode includes the baseband signal (proportional to the "envelope" of the modulated siganl) along with the original signal and its sidebands, and a whole bunch of harmonic distortion at mulitples of the carrier frequency.
It doesn't really matter all that much if you filter out all of that higher-frequency junk or not if you're just driving an earphone with it — you can't hear that stuff anyway because of the limited frequency response of your own ears (and of the earphone, too). All you hear is the baseband signal that the diode created.
The reason that more complex radios use a filter is that those other components can cause problems in any subsequent audio amplifier stages. A relatively simple low-pass filter (usually just a resistor and a capacitor) takes care of that issue up front, leaving you with a clean audio-only signal.