For practical purposes, the most important limit is to go above the acoustic threshold, i.e. the frequency should be more than 20 kHz. Above that it is a matter of mechanical vibrations caused by torque jitter.
Electrically the torque jitter is determined by the motor coil characteristics, most importantly the coil inductance, operating voltage and operating current.
For example, for a small brushless motor the coil inductance could be 100 µH, operating voltage 10 volts and operating current 1 ampere. If the PWM frequency is 20 kHz, the low and high duration is around 25 µs. During this time, the coil current will change by 25 µs * 10 V / 100 µH = 2.5 A. The magnetic field follows the current, which causes the torque to jitter by +- 125%. If the PWM frequency could be raised to 100 kHz, current would change only by 0.25 A and the torque jitter would be +- 25%. Larger motors usually have more inductance, permitting lower PWM frequency.
The velocity jitter is not as large, because the inertia of the motor will smooth it out. But the torque jitter directly causes vibrations in the surrounding structure, and the strength of the vibrations depend on mechanical properties of mass and elasticity.