What is the mathematical relationship between the switching frequency of MOSFET in a three phase motor controller (induction motor and BLDC motor) and the speed in rpm of the motor shaft?
There isn't one really.
The switching frequency of the MOSFETs is a function of the inductance and resistance of the motor and is used to control current ripple (and therefore torque ripple).
The speed of the motor shaft is a function of how fast you change which of the three phases are being powered.
That being said, there are limits. For instance the switching losses incurred from a high switching frequency will limit the amount of power available to the motor and therefore limit the speed with which you can commutate the motor.