An alternative amendment to the circuit is to leave the op-amp's non-inverting input connected to the junction between the two 10k resistors and just reconnect the bottom of all 3 capacitors to 0 V. If its reluctant to start oscillating then you could try increasing the gain a little by reducing that 56k resistor a bit.
The theoretical oscillation frequency for the low pass version is fosc = sqrt(6)/(2.pi*RC) = about 39 kHz.
The actual oscillation frequency will be a little way from 39 kHz because of the lagging phase shift across the op-amp. The oscillation frequency will shift slightly away from 39 kHz in search of the frequency where the loop phase is -360 degrees. If the lag across the op-amp is, let's say, -200 degrees (inversion - 20 degrees) then the oscillation frequency will shift to a frequency where the lag across the three RC networks is -160 degrees.
With the low pass version you can increase the op-amp's gain enough to drive the op-amp into saturation without the spiking problem associated with the high pass version. Driving the op-amp's output into saturation reduces the loop gain to unity and now the Barkhausen Criteria for oscillation have been met (loop gain of 1 and loop phase of -360 degrees).
Take the waveform from the output of the phase shift network and buffer or amplify as necessary with a high impedance input non-inverting amplifier. The output at this point should be a nice low distortion sine-wave as the harmonics from the op-amp's saturated output will have been filtered out by the 3 cascaded low pass filters of the phase shift network.