I noticed that I have a 4-5 seconds delay before the output of my low
pass filter op amp gives the correct mean value of an input square
wave.
R9 and C5 (10 kΩ and 100 μF) produce a time constant of 1 second and, as a rule of thumb, engineers say it takes 5 × the time constant for a signal to settle to 99% of the expected value.
In other words, it is nothing to do with the unity gain op-amp but has everything to do with R9 and C5 and, if you want to heavily filter a signal then you also incur a time delay. The two things are inseparable in the real-world of analogue circuits.
Is it normal for the delay to be that long and if so, are there
recommended alternatives to avoid this delay?
It's normal and there's no alternative; this is what filters do. Of course you can make a higher order filter that takes a bit less time but, at the expense of complication.