What is the determining factor behind how fast a data interface can be?
Several factors. First the Shannon-Hartley theorem states the maximum throughput of a communication channel depending on its bandwidth (BW) and signal to noise ratio (SNR).
The gist of it is: throughput in error free bits/s is proportional to bandwidth, which is quite intuitive.
For short cheap wired links like USB, SNR is usually excellent, so it exits the equation, and immediately comes back in the form of cost savings: when SNR is high, there's no need to pay extra for circuitry capable of digging the signal out of the noise. So you get simple modulation schemes, data is encoded (scrambled, error checking codes added, etc) into 1-bit symbols transmitted as two voltage levels. But then you need a cable with a bandwidth similar to the throughput, in other words you need a cable that will transmit several GHz at low attenuation and low skew (ie, all frequencies arrive roughly at the same time)... and as cheap as possible, of course. Any discontinuity in the transmission line creates reflections, like echoes, so while noise is low, inter-symbol interference can be high. Thus one necessary condition for Gbps USB is to be able to manufacture a complete chain (PCB - connector - cable - connector - PCB) with controlled attenuation and propagation delay over a few GHz, for a total cost of less than $1. It is not easy, so it took a while.
Then you need chips at both ends, fast enough to handle it, for a total cost of 10c or less, and very low power budget too. Again, not easy, so it took a while.
And of course you need to... actually need it, so you need the rest of technology to offer peripherals that can actually use the extra bandwidth, like SSDs.
If you add six zeros to the acceptable cost, and a few zeros to the power budget, not to mention the size, it was doable 30 years ago of course, but the cost was only worth it if you were laying under sea cable or stuff like that.
Basically, as a wet finger in the wind estimation, a USB3 port from 2024 would cost about the same as a USB2 port from 2010 or a USB1 port from 2000. It's only possible to upgrade the performance when it becomes cheap enough.
What's physically stopping my USB 2.0 device that I plug into a 2024 computer from transferring data as fast as a 3.2 device?
USB also has a specific quirk: USB1-2 are half duplex, using one wire pair making one link which is used in both directions, whereas USB3 is full duplex, so it uses 2 wire pairs making 2 links. So they're fundamentally incompatible. The transition to full duplex was needed because the time spent for the bus to "turn around" (ie, change transmission direction) is short but not zero, so at 12Mbps the number of bits not transmitted during this wait time is negligible, but the higher the throughput, the more it matters.
Ethernet is a completely different thing than USB, because cables are much longer. This shifts the cost balance towards making the electronics smarter and more expensive to be able to use cheaper cables, and of course each generation needs to be compatible with the previous one, because replacing cables and sockets is very expensive.
If USB1 had been designed as full duplex, which it wasn't because it would have cost 10 cents more... but if it had been, it would probably have been possible to keep the same USB plugs for USB2 and 3, and have devices at both ends negotiate which version they want to use, just like Ethernet. Of course the 480Mbps cables would have to be upgraded for 4Gbps.