The mathematics behind stability are crystal clear. I am very confused, however, as to how these measurements are taken in a real system. Specifically, say I have a power supply. How would one measure the power supply stability including gain margin and phase margin. I am interested in a controlled way, what kind of probes, etc.
There are ways to measure the transfer function of a given system. However, just to judge the stability a good way is to look at the step response. The overshoot (or the lack thereof) can be used to determine the phase margin of a system.
Often systems can be approximated by a second order transfer function and almost every (basic) book on control theory covers the theory of these systems and the relationship between step response and phase margin.