You need to do what we call a link budget. This typically takes into account:
- the directivity of the antenna/its gain with respect to an omnidirectional antenna (transmitter and receiver)
- the power converted into waves in the antenna. Not to mix up with the power in the transmitter, the efficiency of the antenna driver can vary quite a bit from transmitter to transmitter!
- The free space loss which depends on frequency of the signal (related to baudrate) and distance (absorption of air)
Such a link budget sums all gains and losses and always equals zero, so you can solve for the maximum baudrate, knowing that the signal power at the receiver must be X times bigger than the estimated noise power at the receiver.
X is the signal to noise ratio, which is related to the bit error rate via the modulation technique that is used by the transmitter/receiver. i.e. Certain modulation techniques are more robust against noise than others and don't generate that many upsets.
Usually there is something else on top of the hardware layer which is encoding, to decrease further the bit error rate for a given modulation technique. In its simplest form, it's a parity bit.
With those keywords, you should be able to know what the specs are for bluetooth and calculate the maximum baudrate.
Or you could do it "the experimental way", and run the system for several days at a constant baudrate and record the number of bit upsets. If it's fine to you, stick to it. The above is only for your own understanding, if you can actually test it, test it.
