Your basic problem is that you have an odd image of modulation.
Take your description of a frequency modulation scheme
... the case of FM radio where a 20-20kHz audio signal is modulated by ~100MHz carrier which allows the signal to be transmitted wirelessly ...
That is backwards. You modulate the 100 MHz carrier with the low frequency signal. You seem to assume that the modulating signal must be at a higher frequency than the base signal.
I recommend you read a bit more about modulation, perhaps starting with something simple like amplitude modulation.
The datasheet for the modulated optical sensor you linked to describes the modulation it uses:
How it works: Instead of being on continuously like our other optical sensors, the LED light source in the ROSM is modulated on and off at a high frequency. That high frequency light signal is reflected back at the sensor for processing while all other sources of light are filtered out and/or ignored.
This is a form of amplitude modulation. The transmitter turns the light on and off many thousand times per second. The receiver sees the light and records the intensity. The intensity of the received light pulses many thousand times per second.
This is often used in infrared remote controls. The transmitter turns the infrared LED on and off at 35kHz (35000 times per second.) The receiver detects the intensity of the infrared light. When the infrared detector "sees" the transmitter, a stream of pulses at 35kHz comes out of it. The receiver itself is only interested in the 35kHz pulses. If 35kHz is there then the transmitter is on, else it is off. The transmitter pulses the 35kHz signal to convey the keypress codes. The signal is doubly modulated, if you will. Keypress information onto 35kHz, 35kHz onto infrared.
The modulated sensors are similar.
The transmitter modulates (pulses) the LED with a 35kHz signal. The detector "sees" the pulses, and has an output that says "35kHz is present" or "35kHz is not present."
If you reflect the transmitter signal back to the receiver, then the output will signal "35kHz present."
The interesting bit is that you now aim the transmitter at a rotating object that has a small mirror or white spot on it. When the white spot comes around, it reflects the transmitted signal back to the receiver. As it rotates, it sometimes reflects the signal back and sometimes doesn't.
The output of the sensor changes state as the object rotates. It changes constantly from "35kHz present" to "35kHz not present" as the white spot passes by.
Again, it is doubly modulated. The light is pulsed at 35kHz, then what the receiver "sees" is pulsed by the rotating object.
There are many sources of light that the detector can see. It can't tell light from the LED from light from a light bulb from light from the sun. They all contain red light.
Pulsing the light from the LED makes it distinct. You are no longer looking for just the presence of red light. You are looking for pulses of red light at a specific frequency. You can filter the output of the detector so that only that one particular frequency gets through.
Natural sources of light don't turn on and off thousands of times a second. Light bulbs flicker some, but at a relatively low frequency that is quite distinct from the higher frequency used to pulse the sensor LED.
Pulsing the transmitter signal allows the receiver to ignore all other sources of light and "see" only its own transmitter.
Your sensor may or may not really be using 35kHz. It may be using a higher pulse rate.
Then again, it might just be using 35kHz. That's a commonly used frequency for remote controls and such, so that detector ICs would be readily available. 35kHz translates to 583 pulses per revolution at 250000 RPM, so it ought to be quite detectable.