Consider a classically configured closed loop control system (or amplifier) with a forward transfer function, a feedback transfer function and a summing junction. If this control system is in the theoretical state of being marginally stable then the loop gain is equal to exactly 1 and the loop phase is equal to exactly -360 degrees.
In an ideal world there will be no noise and so, with a grounded input, the output to the system will also be at rest at ground potential. In the real world, the small amplitude wide band noise present at the input to a marginally stable control system will contain a frequency where the loop gain = 1 and the loop phase = -360 degrees and this noise frequency will be "picked out" by the control system with the result that its output will be driven to + & - infinity. (Obviously, in practice, limited by the power rails).
In the case of a theoretical, 2nd order, noiseless, marginally stable system (damping ratio, zeta = 0). If this system is subjected to a step input then the output of the system will oscillate at a constant, limited amplitude for ever.
In the case of a sine wave oscillator (no input or summing junction), a loop phase of -360 degrees (or 0 degrees) combined with a loop gain of greater than 1 results in an increasing amplitude oscillation (instability). Once the required amplitude is reached the loop gain is reduced to 1 (usually by automatic gain control) and then we have a marginally stable system with constant (and limited) amplitude oscillation.
In the case of an ideal integrator, zero input results in a marginally stable system with the output being held at a constant amplitude but even the tiniest of inputs will result in the integrator's output ramping up (or down). Theoretically the output will ramp to infinity but in the real world the output rise will be limited by either the open loop gain or the power rail, which ever occurs first.