Let's start by analyzing what you have designed.
I'll ignore C1 for the moment.
- To hold the Schmidt trigger input low requires the opto to sink 3.3mA, the pull down current for the gate can be ignored.
- The transfer ratio for the opto is about 20% minimum so you need a minimum of 5 * 3.3mA or about 17mA through the LEDs.
- With the resistors you chose for the input this would require a minimum of 3210 * 0.017 or about 57V (55 + LED Vf=1.7max) to supply the current. The voltage at the junction of R3,R1 is going to be about 10.5V.
- The TVS will conduct at about 31.5V minimum, at that point the current through the LED would be about 59mA and the input voltage would have to be about 189V. The peak of the AC mains cycle is only 170V, so the TVS will clearly never conduct with your design. You will also never of course reach the 59mA level into the LED, you will only get about 52mA maximum.
The TVS datasheet shows that the value you used (28.2V) is the largest value where the TVS WILL NOT conduct. The value where the TVS begins to conduct (1mA is the threshold ) is between 31.4 and 34.7V. I used 31.5V for convenience, you could go back and consider best case at 31.4V. However if you consider worst case devices at 34.7V @ 1mA, then you have an even greater discrepancy in input voltage required to cause the device to conduct. Short story. The TVS in the circuit as designed is non-functional, it will never conduct.
As Dave suggested you can increase the value of R2 to reduce the output sink current and therefore lower the input current required, but at lower input current the CTR drops as well. From the LTV-814 datasheet you can see that as you reduce the input current the CTR plummets.
You can see from the above, the 17mA required by your design is about ideal to maximize the CTR, though you should design for minimum CTR of 20% for the spread of devices.
If you use Dave's suggestion and use a 10K Ohm for R2 (a 500uA load), while this reduces the input current requirement you can see the CTR drops and if you look at Fig3 in the datasheet the VCE(sat) rises which means you can't pull the gate input low.
If you set the input current to be say 5mA the CTR has dropped by 25%. If you dropped the input current to 1mA, the CTR has dropped by almost 70%.
My suggestion would be to reset the ratio of R3,R1 so that at the minimum current requirement of the input LED the TVS is about to conduct.
If you keep the input current at 17mA this would mean setting R1 to 1.8K Ohm and R3 to 1.5K Ohm. Double these values and you'd drop the current to 8.5mA and I would not suggest going below that.
The last thing to mention is the value of C1. The opto output does not act like a normal transistor, it acts more as a CC source. This is complicated by the fact that the input drive is a sine wave somewhat, though now the input is being clipped correctly the LED is being driven with a constant voltage/current once the TVS conducts.
The C1 value required will be smaller than you imagine (I'll leave you to work that value out) since the collector current is limited to an essentially fixed value due to the way the opto works. Imagine that the devices you use are all better than the worst case (better than 20% CTR). You might be able to sink 2 or 3 times the current required by R2. However this will still result in a slow charge time for C1.
If you want to play with the front end resistor values, you could use the schematic below, which can be simulated. I've altered the Zener values to make it work since there is no TVS in the schematic editor.
simulate this circuit – Schematic created using CircuitLab