when the Vo>12, which mean the voltage across the Rb is above 2.5V,
the current through the TL431 becomes small, and the Vcomp voltage
No, that's incorrect. When Vo rises above it nominal set-point, then the voltage across Rb rises too and this causes the TL431 to turn on and conduct current through the LED in the opto-coupler.
This then causes the photo-transistor to turn on and lower Vcomp. This then would naturally cause the duty cycle of the PWM to lower and restore Vo to 12 volts. It's a control loop that self-regulates dependant on the resistor ratio of Ra and Rb producing 2.5 volts at the input pin of the TL431.
How to design the Rc3 value in the worst case
You choose Rc so that if the TL431 is turned hard on (i.e. has 0 volts across it), the current into the LED is limited to a value that won't hurt the LED.
Under normal running circumstances it's hard for the TL431 cathode to anode voltage to be much lower than a couple of volts but, for the sake of choosing Rc you should assume that the TL431 voltage could be 0 volts (after all, it may have a capacitor across it in some feedback scenarios so, for the sake of a couple of volts, you should assume the worst case is zero volts so that Rc is chosen so to NOT deliver too much current to the opto LED.)
How do I know the TL431 output voltage.
You don't need to know it - it'll be a voltage that causes the right amount of current to flow through the LED to keep the duty cycle at the right value to maintain the output voltage at the required set-point.
How the TL431 can do the negative feedback.
Negative feedback comes from Ra and Rb. Then through the TL431 (which acts as a voltage comparator). Then through the opto then, into the PWM control pin of the chip that drives the MOSFET (not shown in your diagram). That chip controls the PWM.
All of those components are links in the same chain. That chain makes negative feedback.