Can't we use only a single transistor in emitter follower configuration to have a single-ended class A power amplifier
Sure we can, I guess you mean something like this:
simulate this circuit – Schematic created using CircuitLab
Now imagine that I want a certain output power, that means a certain current needs to be delivered to the speaker. The speaker needs AC, there cannot be a DC current flowing through the speaker so that's why C1 blocks the DC.
Now imagine we need a signal of up to 100 mA to the speaker. That signal needs to be positive and negative. That's from -100 mA to + 100 mA so a total range of 200 mA.
The current through Q1 can only be positive so the solution with the lowest current consumption is to make the current through Q1 vary between 0 and 200 mA. When there is no input signal, 100 mA flows through Q1 and R3. That is what we call quiescent current or biasing current. This current flows even when you turn the volume all the way down! Imagine what this would in a battery powered device. Sure we can lower that current but then the maximum volume would be limited as well.
This high quiescent current consumption is the result of choosing a class A stage. A class-A push pull stage would have the same quiescent power consumption.
The advantage of using a transformer like in the circuit that you show is that you can trade voltage and current. Suppose the speaker still needs 100 mA but I'm using a 10:1 (primary:secondary) transformer. Then at the primary side the current will be 10x lower! OK, the disadvantage is that I will need a voltage that is 10x higher. But with low ohmic speakers (8 Ohm etc) that is not an issue, at 100 mA that would result in 0.8 V. Multiply that by 10 and we get 8 V which is a reasonable output voltage for a supply of 12 V.
So in summary: yes we can use an emitter follower and indeed it does have a low output impedance. However what you didn't think about is that although the small signal impedance of an emitter follower is quite low (1/gm) that does not mean it can deliver enough current to drive a low impedance load like a speaker directly. That's why a transformer is a good idea, it transforms the low impedance of the speaker into a higher impedance that is easier to handle for a low power circuit.