Let us assume that the open loop transfer function of my system is:
\$L(s)=\frac{10(s+1)(s+2)}{(s-3)(s-4)}\$
Also, the system has unity feedback, meaning that the closed loop transfer function is
\$T(s)=\frac{L(s)}{1+L(s)}\$ since \$L(s)=G(s)H(s)\$ and \$H(s)=1\$
Then the Nyquist plot is this:
There are 2 RHP poles present and the plot circles the (-1,0) point 2 times, anti-clockwise. \$Z=N-P\$ where \$N=-2\$ and \$P=2\$ so \$Z=0\$ => Stable System.
And the unity step response is this (for the closed loop):
It looks stable, but if I press on "more time", it looks pretty unstable:
So the Nyquist plot is disagreeing with the unity step response. What is happening?
The transfer function is theoretical in nature, it does not necessarily represent a real system's response. I have taken care to make sure that the unity step response is that of the closed loop transfer function (and not that of the open loop, which is clearly unstable). I used WolframAlpha for all my plots.