The 3.5 mm audio jack is a smaller version of the 6.3 mm phone connector
These 6.3 mm connectors are much more rugged than any USB connector. For example they're round so they have no up/down side. You just plug it in, done. Ideal for musicians and telephone operators (as suggested by comment by Henry Crun) which have no time to fiddle with connectors.
Also audio does not need so many connections, 3 is enough for a stereo headphone.
The 3.5 mm audio jack was designed for portable equipment, the 6.3 mm concept worked so they just made it smaller. Smaller also means less rugged so more fragile. A smaller connector would also mean introducing problems after extended usage.