I understand, that 802.11x protocolls can use different type of modulations. BPSK, QPSK, 16-QAM, 64-QAM... I also understand that the used modulation depends on the quality of the channel, I guess on the SNR. But how does the actuall process look like? How does the transmitter decide what modulation should be used?
As usual, it depends.
Let's restrict ourselves to the OFDM-based IEEE802.11 standards (a,g,n,ac,p, and many more).
These have a typical frame structure, in which one symbol, which is always BPSK-modulated, contains the rate setting info (and thus, implicitly, the modulation used).
As an example: Got this graphic from keysight
See the violet symbol.
Notice that in OFDM, you can use different modulations for different subcarriers, or in fact, you can freely pick your modulation for each frame and subcarrier. You don't typically do that in Wifi (the standards restrict your freedom here, to varying degrees, depending, again, on the individual 802.11 standard); in 4G/LTE OFDM systems, you can expect a frame of the downlink signal to be aimed at a whole set of different receivers, which all pick out "their" part in the subcarrier/symbol plane – and thus, you can expect different robustnesses to apply to different ressource blocks. Yay, complexity!
In essence: for wifi-style OFDM 802.11, the transmitter decides on what modulation it uses for the payload data itself. The optimal choice depends on a lot of factors, some of which are receiver capabilities, throughput and power consumption limits, and an estimate for the interference and noise levels at the receiver.
Thus, the algorithms picking an optimal modulation for any specific instant in any specific use case for any specific x in 802.11x are very diverse, and, often, very highly optimized by the chipset/driver vendor, and thus can be very obscure.
Modulation and Coding Scheme (MCS) indices determine the data rate in 802.11, based on the SNR observed. Each index uses a pre-defined modulation scheme (upto 256-QAM in VHT modes) and a coding rate (upto 5/6), which in turn influences the data rate.
Note that higher your data rate, higher would be the SNR requirement. This should give you a neat idea.