You mention high time constant (very low cutoff frequency) filters. Others have made nice answers about capacitor ESL/ESR and high frequency effects, so I'll focus on low frequency stuff.
First, you got to decide what precision you want on the cutoff frequency and overall response of your filter. High value caps (like electrolytics) usually have extreme tolerances, like -20/+50%.
High-K ceramics like X7R change value with temperature (by a lot!) and also with voltage (which creates enormous distortion in your signal). They are also excellent microphones due to piezoelectric effects. Never use X7R/Z5U and the like in any kind of precision analog stuff! They are designed for decoupling, and excellent in that role.
Electrolytics' ESR varies with temerature, too, and the capacitance tolerance is huge. And there is dielectric absorption, which will screw your filter response at very low frequencies.
Only film and C0G ceramics types have reasonable precision, low temperature coefficient, and are generally what we would consider as "stable" in a wide, usable temperature range.
So, if you don't care about frequency response accuracy and only want a DC-blocking cap, then you can use electrolytics. In a filter, you'll have to worry about ESR, leakage, and dielectric absorption.
If you do care about accuracy, then your only choices are C0G/film types, which are only available in low capacitance values. Film caps are also expensive.
And large thru-hole film caps are microphonic, too, and subject to dielectric absorption... unless you use polystyrene or polycarbonate... which is even more expensive.
This is why you're going to implement your filter using one of these methods:
- Use FET input opamps and large resistor values, which allow smaller caps, which allows you to choose the good caps: C0G, PPS, polystyrene, polycarbonate, polypropylene... Most practical solution really...
- Obsess over low noise, pick an ultra low noise BJT input opamp, then pick low resistor values because of input current noise, then pick huge caps, then realize too late they are unaffordable and microphonic.
- Worst case is when you need to pass HF signal, but also filter LF out. This will get you a fast path with a fast opamp, and the lowpass implemented as a DC servo with a FET opamp.
- Sample the signal and use a digital filter. If you need low cutoff frequencies, then a slow sampling rate is OK, a micro implementing an IIR filter will do the trick. Of course you need to worry about rounding errors and quantization noise now. If your filter is placed before an ADC, then you should decide which parts of the filter you should implement in analog, and which parts in digital.