The main technical difference is how they reject interference. Twisted pair relies on the interference affecting both wires equally, generating common mode noise that is easily rejected by the differential receiver. This works well for magnetic interference down to very low frequencies.
Coax cable relies on the magnetic interference inducing opposing currents in the shield that cancel out the magnetic field inside. Penetration of the magnetic field into the cable is limited by skin effect. This works well at RF frequencies, but poorly to useless at audio and power line frequencies. At 50Hz the skin depth is ~9mm, so the interference goes right through the shield.
So which is best depends largely on the frequencies involved and the type of interference that may be present, but it's not the only reason to chose one over the other.
Analog telephone lines often have to run close to power lines over long distances while carrying fairly low level audio signals. The human ear is quite sensitive to power line harmonics that coax would would not be able to reject. Coax cable is also bulkier and more expensive, which is a big deal when you have to run thousands of them over many kilometers. Imagine this, but with 1800 individual coax cables bundled together...
Twisted pair can also work well at higher frequencies, but the cable dimensions may be inconvenient. TV sets used to use 300Ω 'ribbon' cable, which actually has lower loss than standard coax at VHF frequencies. But it was annoying to use because it had to be kept away from the metal roof etc., was prone to weather damage, and a balun was required to convert to 75Ω unbalanced at the receiver.
At higher frequencies coax has the advantage of lower loss and wider bandwidth in a robust cable with excellent shielding, and the unbalanced signal is easier to interface to. Cable runs are generally short so cost isn't so much of an issue - except for CATV, but then (unlike telephones) each subscriber doesn't need their own circuit so a single cable could serve thousands of viewers (modern CATV is mostly fiber optic so coax runs are much shorter).
Coax cables are commonly used in audio to connect between components and inside equipment, despite not being very effective against low frequency magnetic interference. However the circuit impedances are generally in the 1k to 1M range so magnetic interference (which generates high current but low voltage) is less of an issue. The coax still protects against electric fields (which have more effect at higher impedance) and rf interference of all types. Low level audio signals may need better protection, and then shielded twisted pair is often used. This combines the advantages of both cable types.
I can see that the 50 Ohm concept is good to get rid of reflections in
transmission line theory. But how come the unbalancedness of coaxial
cables causes no problems with impedance balancing issues?
Balanced or unbalanced makes no difference for impedance matching, and precise matching isn't always required anyway. If the cable length is much shorter than the signal wavelengths then reflections aren't a problem in most applications. Nobody cares about coax impedance in audio applications, and even composite video (with a bandwidth of ~6MHz) isn't visibly affected by unmatched cables in equipment cables.