what is the price you pay for this bit rate increase?
The price you pay according to Shannon-Hartley Theorem is SNR. You need a much higher SNR (S/N) or Eb/No ratio to compress more Bits/baud.
Eb/No QAM [dB]
16 13.0 (+3.5)
64 17.5 (+4.5)
256 22.5 (+5.0)
For Gaussian Noise you can ...
What happens when you increase the amount of distinct states in QAM is you get a higher BER (bit error rate).
To achieve the higher data rate with the same BER, you need to use error correction coding. You see, the BER rises, but your ability to correct the errors increases at a greater pace, as Shannon discovered.