According to this source:
Digital modulation techniques are classified as linear if:
- The amplitude of the transmitted signal varies linearly with the modulating digital signal, \$m(t)\$.
- They usually do not have a constant envelope
- They are more spectral efficient.
- Poor power efficiency
The only other "linearity"-reference I could find was in the same course slides:
For Frequency Modulation, the relationship between received power and quality is non-linear (Rapid increase in quality for an increase in received power).
For Amplitude Modulation there is a linear relationship between the received signal quality and received signal power.
But I am not totally convinced that this is at the basis of the definition (I'm also not totally unconvinced).
IMHO, I think they started out calling the AM scheme linear, as \$AM(a\cdot V_{in}) = a\cdot AM(V_{in})\$, and then they started playing with the phase ((Q)PSK) and just labeled it as linear as well, while it isn't strictly true.
[EDIT] I also found this which seems to support my last remark.