Question
How come my GPS module takes so long time to fix, after TTFF (Time To First Fix)?
Answer
Update - I now realise my answer below will apply to many similar GPS signal acquisition problems but does not answer the OP's question in this specific case.
The OP asks why first fix is reasonably long, but later fixes are disappointingly not that much shorter.
@Marko Buršič already gave the correct answer, with the reason that no battery backup is the root cause of the problem.
End of update
Note - As explained above, my answer below is not the correct answer, but can provide some background knowledge for newbies to understand the situation better.
Now let me share my GPS testing and fixing experience. My experience is about 100 hours learning and testing, with 5 of so different GPS/GNSS modules.
Yes, sometimes it takes 3 to 5 minutes to get the first fix TTFF (Time To First Fix). But almost always, afterwards it takes much shorter to do subsequent fixes, in average 30 seconds or so (shortest 15 seconds, longest 2 minutes).
I always use my GPS module at home, near two windows facing two directions. I found that I need to extend the antenna at least 15cm from window, but no need farther, even my open sky is not that open (see Appendices for some photos). I found the optimum distance is just some two meters from window.
I surprisingly found that the "openness" of an open sky does not matter that much. Often I just hang my GPS module outside the wall of my flat, some 20 stories high, with a one meter long connecting wire to the active antenna.
I also surprisingly found that power supply noise does not mater that much. Using a Lipo battery power bank, hoping to get little noise, and best performance, does not help at all.
I once used GPSD with ATK1218-BD (BeiDou), but the learning curve very steep and not friendly to newbies. One the other hand, for NEO-6m/7M/8M, I found uBlox uCenter "IDE" very good. Tip - SparkFun has newbie GPS tutorials.
I have also written basic Rpi4B Thonny python 3.7.3 programs to extract GPS data, based on uBlox uCenter. I found everything straight forward and easy (Of course the GPS NMEA1083 protocol is already text based and easy to play with python.)
For more GPS setup and testing details, you might like to read my answers to similar GPS questions in the reference list below.
References
(1) Rpi Neo-6M / Neo-8M GPS Module Setup / Fix / Update Problems
(2) How can Rpi listen to a GPS module?
(3) How can Rpi connect a GPS module?
(4) The Basics of GPS - SparkFun
(5) How GPS Works (YouTube 12 min watch) - SparkFun 2017jul31
Appendices
Appendix A - Photos of my open sky at home
Appendix C - YouTubel showing a GPS module blinking once per second after fixing
Video showing GPS module blinking once per second after fixing
Notes
(1) The little backup battery that the OP must feel jealous is at the left side the blinking blue LED.
(2) The tiny connector connecting the black wire to the active anetnna is tricky to connect, must read SparkFun for a newbbie tutorial on how to connect.
(3) Even if you have learned how to connect, be warned that the number of connections allowed is only 5. That means after 5 times plugging in, the contacts will become loose and very easy to fall out, so you see I am using cable ties to fix the cable in secure position to "release strain" caused by careless guys like me pulling or tripping long antenna cables.
And if you find fixing intermittently fails, it is probably the poor contact connector due to you poor craftsmanship, so don't curse half of the Universe for its clear blue sky, or the ridiculously packed match box size flats blocking the sky waves. And yes, I also live in a match box, or what we locals called a "coffin" room, and I am playing GPS toys inside a coffin, and I am loving it :)
Appendix D - Crowded Buildings
/ to continue, ...