This is only partially related to the BackEMF of an electrical machine.
Assuming you have a 4-quadrant controller & you are ACTIVELY decelerating the rotor, what is occurring is the transfer of energy from the rotor ( \$\frac{1}{2}J\omega^2\$ ) and the stator inductance ( \$\frac{1}{2}LI^2\$ ) onto the DCLink capacitance (\$\frac{1}{2}CV^2\$)
The voltage on the capacitance must rise as the energy is transferred.
Three ways to minimise or deal with the increased voltage
- Increase the DClink capacitance close to the H-Bridge.
With an increase in capacitance the final voltage for the same energy transfer will have been reduced
- Decrease controller bandwidth
It is not stated whether you have a form of PI closed loop control, nor whether you have a speed loop, but reducing its bandwidth and the rate it can decelerate will reduce the rate of energy tranfer which will reduce the end voltage
- Incorporate a resistive brake circuit
By placing a Resistor + FET across the DClink (plus a freewheel diode across the resistor) & a DClink hysteretic monitor (comparator), the FET will "chop" the DClink between predefined thresholds (say .. 54V -53V). As long as the resistor value and power rating have been appropriately selected, the DClink will be maintained below the level of concern