Is there any relatonship between stability and overshoot? I mean, can we say that a system if it is marginally stable, it will not have overshoot?
This is a complicated subject but one can try to give some guidelines. The below picture shows the typical open-loop gain response of a switching converter: the plant response \$H(s)\$ is of 2nd-order while the compensator \$G(s)\$ is of 3rd-order. To reduce the order and simplify the analysis, the system is observed in the vicinity of the crossover frequency \$f_c\$ and is approximated to a second-order system, without any zero:
What is interesting now is to see how the open-loop phase margin that you select affects the closed-loop quality factor \$Q\$. In other words, how can I can select my phase margin to shape the transient response I want. Because this is really the point: you choose the phase margin based on the wanted transient response. If you do the maths as described here, you end up with a simple relationship linking phase margin and quality coefficient:
In this picture, you can see that going to a large phase margin (thus an extremely stable converter), brings a low quality factor meaning a non-oscillatory response, without overshoot. On the opposite, should you accept some overshoot because it also goes with response time, then you would purposely adopt a less conservative phase margin. When the phase margin is 76°, the quality factor is 0.5 implying coincident poles: the response is fast and non-ringing as the roots are real.
Below is the typical step response of a switching converter whose crossover frequency is constant to 5 kHz but the phase margin is changed from a sluggish non-overshooting response (90°) to a fast-recovering but ringing waveform when the margin decreases.
Please note that this is a purely theoretical approach in which we have modeled a 5th-order transfer function featuring zeroes as a 2nd-order polynomial denominator. This is to explain how phase margin affects the response and overshoot in particular.