I understand return loss is the ratio of how much signal is reflected in a transmission line. But for wireless devices there is no cable. So how is this measured?
So is the impedance mismatch for wireless devices between air and the antenna?
A transmission line can be a cable.
But a transmission line can also be air (or vacuum).
To get a signal (from a cable) into the air we would use an antenna. The reverse is also true, to get a signal out of the air into a cable also an antenna is used.
An antenna is basically the device that "connects" the cable to the air.
If this would all go perfectly (no or little losses) then all the signal from the cable ends up in the air (or vise-versa).
But if there is a mismatch than (part of) the signal will reflect, this is what your question is about.
This mismatch can be measured from the "cable side" of the antenna. Instead of connecting it to a transmitter or receiver, we connect it so a network analyzer or a vector impedance analyzer.
This is a device capable of making a plot of impedance over frequency, reflection coefficients (S-parameters, for your question it is S11). There are also network analyzers that can make a plot of the characteristic impedance as a function of the distance the signal has traveled. That can be used to find where a cable is broken but also where the mismatch is in a cable to antenna connection.
Here's an example of a plot of the characteristic impedance of as a function of the frequency. We can see that the system reflects the least at 871 MHz. At 871 MHz the of the 0 dBm input signal only -24.29 dBm reflects back. Obviously this system is to be used at 871 MHz!