Note that the LMC6772 chip you are using is not an opamp; it's a comparator that has an open-drain output. Since it can't actively drive its output high, the Schmitt trigger circuit won't work as designed.
The datasheet says, "Refer to the LMC6762 datasheet for a push-pull
output stage version of this device."
However, look at how you have the optocoupler wired up: the anode of the LED goes to the output of the comparator, and the cathode goes to ground. This means that when the comparator output is "high", the LED is driven by the current coming through your resistor network, and when it is "low", the diode is shorted out. This also means that the "high" voltage at the output of the comparator is the forward voltage of the LED. (According to the datasheet, this is at most 1.8V)
Do NOT switch to the push-pull version of the chip unless you also add a current-limiting resistor in series with the LED.
Note also that the website you used to calculate your resistor values is assuming that the opamp in question has bipolar supplies, and that the output swings to -Vcc when the output is "low". Your opamp is being powered by a singled-ended supply, which makes the lower threshold 7.56V when the output is low.
So, the actual threshold voltages of your circuit are 7.92V when the output is at 1.8V and 7.56V when the output is at 0V.
The general equation for the the voltage at the junction of three resistors, each fed by a voltage source is:
$$V_{junction} = \frac{\frac{V_A}{R_A}+\frac{V_B}{R_B}+\frac{V_C}{R_C}}{\frac{1}{R_A}+\frac{1}{R_B}+\frac{1}{R_C}}$$
In your circuit, one of the three resistors is always connected to ground, so you can ignore that term in the numerator. If we plug in the voltages and resistors you used:
$$V_{junction} = \frac{\frac{12 V}{150 \Omega}+\frac{1.8 V}{470 \Omega}}{\frac{1}{150 \Omega}+\frac{1}{470 \Omega}+\frac{1}{560 \Omega}} = 7.92339 V$$
$$V_{junction} = \frac{\frac{12 V}{150 \Omega}+\frac{0 V}{470 \Omega}}{\frac{1}{150 \Omega}+\frac{1}{470 \Omega}+\frac{1}{560 \Omega}} = 7.56141 V$$
But you're right, these thresholds should work fine in your application, and they don't explain the problems you're seeing. I can only surmise that you're seeing some secondary effects arising from the fact that you're trying to operate a "micropower" comparator at some fairly high voltage and current levels.