The general idea is that you want to recover the modulating signal with as little distortion as possible. There are two sources of distortion: the "contamination" by the remnants of the carrier signal, and the amplitude and phase changes introduced by the R-C filter itself. The minimum overall distortion occurs when these two effects are approximately equal.
I'll leave it to you to demonstrate that this occurs when the R-C cutoff frequency is approximately equal to the geometric mean of the carrier and modulating frequencies.
In practice, the amplitude changes caused by the filter are not really all that important, so it generally makes sense to set the R-C cutoff value to some small multiple of the baseband signal's bandwidth, regardless of the carrier frequency.