Often, if older standards become obsolete, it's because they are superseded by newer technologies. In the past networking was done using coax, instead of the twisted pair used today. Why did they use the more expensive coax? It doesn't look like the twisted pair technology didn't exist back then, so technological advances don't seem to be the reason.
Coax was used for its controlled impedance, its bandwidth and its self-shielding properties.
Sure, twisted-pair wiring has existed for a very long time, mostly used to carry audio frequencies in telephone wiring. That isn't where the technical advancement was required. In order to compensate for twisted-pair's lossiness and impedance issues, major technological improvements in the electronics used to interface to it (such as high-speed adaptive equalizers) were required in order to make it more cost-effective than coax.
Unshielded twisted pair (CAT1 and CAT2 Phone cables) at the time (early '80) did not have enough performance, and coax (10Base2, 10Base5) did not require hubs (which were quite expensive then). When 10BaseT was ratified (CAT3), the price of hubs (and switches afterwards) went down due to mass production.
The problem is not with the cabling. The problem is with the relative cost of electronics versus cabling. Twisted pair existed at that time, as it had been used for phone lines for a long time.
However, the idea of having a separate piece of hardware doing switching in the middle, using the star topology used today for twisted pair, would add cost and complexity to a technology that was already pretty complex. Running a coax cable to each of the machines, and treating it like a true medium, meant you only needed to install a single network interface in each computer. At a time when ICs and transistors and PCBs were relatively speaking more expensive than cabling, not using a central hub/switch saved cost.
Unshielded Twisted Pair has existed for as long as phone lines have existed, but it was originally designed to reject noise at audio frequencies. It was not designed with the correct number of twists to reject noise at the high speed digital frequencies of Ethernet. Coaxial cable, however, had long been in use for various RF applications, and its properties of noise rejection at those frequencies were well understood.
Furthermore, existing coax could allow for interconnect distances from 0.2 km up to 0.5 km. The ability to transmit over long distances without requiring an expensive repeater are crucial when wiring a large building or campus, which were their original design targets.
The first engineers who invented Ethernet technology simply turned to a cabling product that they knew could reliably do the job. Make it work first, then worry about making it better later.
Twisted pair didn't exist in 'the beginning'. There was thick coax (10base5) later thin coax (10base2) and even later UTP/FTP/STP (unshielded, foiled, shielded twisted pair) (10baseT, 100BaseT, 1000BaseT, 10GBaseT, ...).
Twisted pair is much cheaper than coax and it allowed for a star architecture rather than a bus architecture. The advantage for a star architecture is that if a cable fails (or a terminator), only one single device is impacted and the rest will keep working. And administration is much simpler.
The installed base for UTP (structured) cabling is huge and installing cables in buildings is often a major part of the cost of the network. So companies demand network vendors to keep developing equipment that can be used with the (structured) UTP cabling that is already available. Today you see a move from wired network to wireless networks (WiFi), many new office buildings only have very limited datacom cabling installed, often only to central locations from where the WiFi access points (and a lost telephone) is fed.
Unsure if the original ICS cabling for Token Ring was an early variant of twisted pair (probably was), but it was certainly thicker and had fewer leads than what we know as UTP.
And so on May 22, 1973 Ethernet was born. After months of effort built on Metcalfe’s ideas and Boggs’ help in designing and debugging the necessary network hardware, the first working Ethernet prototype, a 2.94 Mbps CSMA/CD system connecting more than 100 workstations on a 1 Km cable, went live on November 11, 1973. Based on its demonstrated success, Xerox would go on to patent Ethernet in 1975.
In 1973, dial up modems, using twisted pair, were 0.0012 Mbps or 0.0003 Mbps, and 2400 (0.0024 Mbps) was the expected maximum achievable. Each twisted pair supported only 2 workstations, one at each end. An advantage was that at those speeds distances much longer than 1 Km were achievable over the dial-up network
So while it is true that "twisted pair technology existed back then", it is misleading to compare it to early Ethernet.
Another important development that allowed UTP to be used for low voltage signals is the cost of transformers versus IC's used for differential balancing thereby significantly reducing the shielding and isolation requirements. All of that iron and copper was expensive to acquire as raw material and ship as finished product.