In order for electrolytic capacitors to perform correctly, they need to be forward-biased. You cannot reverse-bias them and have them perform properly.
Even though your input signal has a DC component, your feedback capacitor is still going to spend some of its time with its polarity reversed as the op-amp attempts to follow the filtered input signal, which means the feedback capacitor won't perform as a capacitor. Instead, it will function as a capacitor half of the time, and an ever-hotter short-circuit the other half of the time. It won't blow up as quickly as if you attached it in reverse to, say, a 12V power supply, but it won't last very long, and your circuit won't perform as expected. What will probably happen is that your voltage regulator feeding the op-amp will get very hot and burn out before the capacitor boils. So, you cannot use an electrolytic for the feedback capacitor.
An electrolytic may work for your capacitor connecting the positive differential input to ground, assuming your input signal is centered at some DC component. But, if you modify this circuit and remove the DC component of the input signal, and feed the op-amp with two supply rails, one positive and one negative, then you can't use an electrolytic capacitor in either position.
It's probably safest just to go with non-polar capacitors, rather than mix-and-match.