You mention high time constant (very low cutoff frequency) filters. Others have made nice answers about capacitor ESL/ESR and high frequency effects, so I'll focus on low frequency stuff.
First, you have got to decide what precision you want on the cutoff frequency and overall response of your filter. High value caps (like electrolytics) usually have extreme tolerances, like -20/+50%.
High-K ceramics like X7R change value with temperature (by a lot!) and also with voltage (which creates enormous distortion in your signal). They are also excellent microphones due to piezoelectric effects. Never use X7R/Z5U and the like in any kind of precision analog stuff! They are designed for decoupling, and are excellent in that role.
Electrolytics' ESR varies with temperature, too, and the capacitance tolerance is huge. And there is dielectric absorption, which will screw your filter response at very low frequencies.
Only film and C0G ceramics types have reasonable precision, low temperature coefficient, and are generally what we would consider to be "stable" in a wide, usable temperature range.
So, if you don't care about frequency response accuracy and only want a DC-blocking cap, then you can use electrolytics. In a filter, you'll have to worry about ESR, leakage, and dielectric absorption.
If you do care about accuracy, then your only choices are C0G/film types, which are only available in low capacitance values. Film caps are also expensive.
And large thru-hole film caps are microphonic, too, and subject to dielectric absorption... unless you use polystyrene or polycarbonate... which is even more expensive.
This is why you're going to implement your filter using one of these methods:
- Use FET input op-amps and large resistor values, which enables the use of smaller caps, which further allows you to choose the good caps: C0G, PPS, polystyrene, polycarbonate, polypropylene... This is the most practical solution, really...
- Obsess over low noise, pick an ultra low noise BJT input op-amp, then pick low resistor values because of input current noise, then pick huge caps, then realize too late they are unaffordable and microphonic.
- Worst case is when you need to pass HF signal, but also filter LF out. This will get you a fast path with a fast op-amp, and the low-pass implemented as a DC servo with a FET op-amp.
- Sample the signal and use a digital filter. If you need low cutoff frequencies, then a slow sampling rate is OK, a micro implementing an IIR filter will do the trick. Of course you need to worry about rounding errors and quantization noise now. If your filter is placed before an ADC, then you should decide which parts of the filter you should implement in analog, and which parts in digital.