The design of the circuit is such that the non-inverting input lags behind the inverting input due to the delay charging and discharging C2. This allows the circuit to compensate for changes in background IR levels, but if, for example, the '-' input were to rise suddenly the '+' input will lag behind while rising with the typical RC time constant and the output will switch low during this time because '-' input is greater than '+'.
The time constant of the delay is given by \$ \tau = R \cdot C = 100k \cdot 1\mu = 0.1 s \$. In \$1\tau\$ it will have caught up by 63%, in \$ 3\tau\$ by 95% and in \$5\tau\$ by 99% of the change.
After the '+' input catches up the op-amp output depends on the offset voltage of the op-amp and it will be very susceptible to noise on the inputs and may, as you discovered, fluctuate between high and low.
To fix this you need to decide what output you want when the circuit is stable.
simulate this circuit – Schematic created using CircuitLab
Figure 1. Pull-up or pull-down resistors.
- By closing SW1 and gently biasing the '+' input slightly high OUT will normally be high. Only when Vin increases suddenly (above the '+' input) will the output switch low. After several time-constants the output will turn low again. No switching will occur when Vin decreases.
- By closing SW2 instead the opposite action will occur. The output will be normally low and a sudden negative step in Vin will cause the output to go high for a time.
You haven't given any figures for variation in Vin when the photo-diode is switched so I've guessed at 4.7M bias. Note that the strength of the bias depends on the Vin voltage. e.g., If Vin is normally around only 1/5 of supply then then R3 will have much greater 'pull' than R2 would. You'll have to experiment.