Consider a negative feedback system that is initially operating in open loop. Apply a sine wave signal and adjust the frequency until the phase shift of the output sine wave relative to the input sine wave is -180 degrees. (Note, this is not always possible, and usually means that the closed loop system cannot be unstable.)
Now examine the gain, \$\small G\$, of the open loop system at this specific frequency. Three possibilities exist: \$\small G=1\$; \$\small G<1\$; or \$\small G>1\$.
Consider the \$\small G=1\$ case; this gives an output sine wave of exactly the same amplitude as the input, but -180 degree phase shifted. In other words, it's upside down compared to the input.
Now connect the output sine wave to the minus input of a comparator. This turns the output sine wave upside down again, i.e. it's now exactly the same as the input. So we could remove the input sine wave and use the output sine wave instead. Essentially we have closed the loop, and the system is now in negative feedback mode and is providing its own input signal. This is critical stability and the system is an oscillator.
Going back to the three possible gain values:
If \$\small G>1\$ at the critical frequency then there is more than enough amplitude in the output sine wave to maintain a constant amplitude oscillation - the closed loop system will be unstable, since the sine wave is continually amplified as it goes around the loop.
If \$\small G<1\$ at the critical frequency the output amplitude is less than the input, so when the loop is closed a constant amplitude output cannot be maintained - the closed loop system will be stable.