This Sigma Delta Modulator is primitive and you are pushing it to find its limitations.
One is that the the Bode plot of the closed loop gain has almost Q of > 8 at 20kHz and you are finding that 10kHz 1mV induces distortion and resonance at 200kHz from impulse corrections.
The result is that your input RC filter of 1k/0.1uF LPF is loaded and pushed back by the negative feedback to a resulting LPF breakpoint of 20kHz with a gain of >8.
If you can do Bode Plots, this analysis becomes easier but the output impedance of the 2nd OA and gain, GBW , current limits of the 1st will all affect these results.
Without me doing your homework on Bode Plots , Do these simple edits.
- change 100nF filter to 2nF
- add 5 Ohms series to the 15nF cap. ( phase lead/lag compensation)
Compare results. It should have minimal overshoot to a square wave input and a higher commutation/decimation frequency.
p.s.
A 4th yr EE student of Control Theory might find the equations of the transfer function and with these changes compare it now to a PID filter with Integration on the 1st Low pass filter, Differentiation on the feedback cap and the 5 Ohm low R , low proportional gain feedback. Hence PID loop compensation but with a decimation comparator front end for Delta modulation. The 5 Ohms also limits the current and rise time of large inputs and causes hysteresis , so this change is not optimal for large swing. Hence a primitive delta modulator.
But the lesson learnt here is to experiment with value changes to find the tradeoffs between small signal response and large signal response, from current limits or slew rate limit or GBW limits and get a better understanding of the intracacies of this circuit then look up the concepts I suggested to learn more.