You want to put a low-pass filter on any ADC input (except the very high-speed types which are being used to acquire very high-speed signals). 60 Hz pickup is not the only source of noise you have to worry about. Line noise from nearby motors is another, not to mention radio stations and such. The reason you have to worry about such high-frequency sources is that any transistor junctions can act as rectifiers, and even radio signals can occasionally show up on low-frequency ADCs due to this effect. The less often you sample the signal the more important this is.
Input filters can be thought of as dealing with two different sources of "noise". The first is aspects of the original signal which are simply beyond the Nyquist limit, and in this role is referred to as an antialiasing filter. The second is injected noise which is beyond the abilities of the circuit handling it. In this aspect, it's particularly important to provide filters at the point where the external signal, such as the inputs from a temperature sensor, enter the electronics. This applies to signal conditioning amplifiers especially.
So there are really two filters to think about - system input and ADC input. The system input filters should be capable of rejecting very high frequencies - think ceramic capacitors with fairly low values. The ADC filters don't necessarily need to be separate from the signal-conditioning opamps - you can incorporate them into the gain stage, usually with just a single capacitor added across the gain-setting feedback resistor.
Of course, you don't want to over-filter the signal and lose information, so you need to pay a little attention to exactly what frequencies are important to you. Another possibility is to sample your signal at a higher rate than you first think you need, and then perform a lowpass filter in software. This is not necessarily a big deal, since a simple running average will make the equivalent of a single RC section. I notice you've done exactly this. What you don't realize is that in order to get a lower filter frequency you have to perform the running average over a longer string of data. You might try playing around with this. Note that this can be computationally efficient, but at the cost of increased intermediate storage. That is, if you allocate a section of memory to your raw data, you can keep a variable as the running total with the new data being added on a sample-by-sample basis at the same time that the oldest data is subtracted from the total. After the cycling through enough samples the total will have flushed out all the invalid sample data, and you're good after that.