So, now I've convinced you (hopefully) of the need for an anti-alias filter, let's look at how to go about specifying and designing one with a simple example. Let's take an easier problem than your photo sensor example to show the basic process, (or one approach to solving this).
Let's say you have a circuit which is battery operated, at a nominal 12V, but that voltage can vary between 10.5V and 14.5V. This battery supply feeds both a high-frequency buck converter to drive your microcontroller, and some actuator whose drive must be altered to compensate for changes in battery voltage as it discharges, to an accuracy of ±5%. So you need to measure the battery voltage with a 12-bit ADC in your microcontroller, which has a 2.0V reference, and which you can set up to read at 1ms intervals to an accuracy of ±1%.
The simplest approach is to feed the ADC pin with the output of a potential divider. The ADC requires a source impedance of less than 3k, so we can use a potential divider with a 3k3 to 0V and 22k to Vbat will give us a good resolution, zero offset and a range up to 15.3V. It also gives a source impedance of 2k87 which is just below the ADC limit. But we can get the same division ratio with 2k7 and 18k which gives better impedance margin at 2k4. So we'll use that.
With standard 1% resistors, the potential divider contributes nearly ±2% of uncertainty, making a total of ±3% for the whole measurement so far. So we have ±2% left for residual aliasing.
Now, the Nyquist theorem requires that your sample frequency must be at least twice the highest frequency in your signal to prevent aliasing. Put another way, and involving the residual tolerance derived above, we must make sure that any component of the signal likely to alias makes no more contribution than ±2% of full range, or ±300mV.
The buck converter we have chosen is used in another battery powered product, and while it gives good output stability, low loss and low component cost, we know it tends to inject around 900mV of p-p noise at 130-180kHz into the battery supply, which is otherwise very stable. Because our application is not identical, we decide to design for a worst case of 1.5V p-p, (±750mV) at 120-200kHz.
This is a particularly nice scenario, because we have a defined signal with the potential to alias, and that signal has a very high frequency, compared with the sampling frequency. If we knew nothing about the potential interference, we would have to design a filter to reject all frequencies above 500kHz, (half the sampling frequency), which is a much tougher challenge.
So now we need to design a filter which will reject ±750mV at 120kHz, leaving less than ±300mV. That's a doddle!
If you recall any filter properties, you might remember that a first order filter, (like a single RC), attenuates frequencies above the cut-off frequency, and that every doubling of frequency beyond the cut-off frequency halves the signal amplitude, (or by a factor of 10, 10dB, per decade). [Note that signal power drops off twice as fast, because power is proportional to amplitude squared].
Now, we need <300mV from a 750mV signal, which is <0.4. So our cut off frequency only needs to be <0.4 x 120kHz, which is 30kHz. But it is only sensible to have a cut off frequency no higher than half the sample frequency, so we could drop in a 500Hz filter which would attenuate the buck converter noise to 750 x 500 / 120,000 = 3mV.
We can achieve a cut off frequency of <500Hz simply by adding a 220nF capacitor across the 2k7 bottom resistor of the potential divider, (the cut off frequence Fc = 1 / (2pi RC), where R is the 2k4 parallel impedance of the potential divider. Simples!
Does that help further?