I'm designing an envelope generator for a sort of mini synthesizer I'm currently working on. I want the envelope signal to evolve (rise or decay, depending on the SPDT switch) only when the switch (SW1) is closed. The first part of the circuit (RV1, C1, R1 and D1) handles the rising envelope signal. The first op-amp ensures that the capacitor discharges only when the switch is open (through D1 and R1) and buffers the signal. That signal is then inverted to generate the decay envelope signal. The problem is that, when the switch is open, the output of the first op-amp, which should be very close to ground (typically 5mV according to LM324 datasheet), is at about 600mV. It surely can't be an impedance problem, the output impedance of the buffer is very low and the input impedance of the inverting amp is 10k. However, if I swap R2 and R3 for 100k resistors, the offset is minimized (I get about 60mV instead of 600mV). Why does this happen? Does it have to do with the inverting amplifier being biased at 2.5V?
Edit: LM324 power supply voltages are 0V and 9V (9V single supply).