The control loop in your example uses a TL431 which is a self-contained op-amp plus reference. This part needs at least 2.5 V to operate properly. When you reduce your output voltage to 4 V, you are forced to reduce the LED series resistance to a low value (100 ohms in your example) to ensure regulation considering the 1-V drop of the optocoupler LED (see my book on loop control). This LED series resistance sets the mid-band gain of your compensator. And with such a low value, you have a very large crossover at a 4-V output compared to the one you have at 34 V. And in this high-frequency region, the phase lag is already well pronounced hence the poor phase margin.
One way to overcome this is to decouple the fast lane from the slow lane. Please read this seminar that I built some years ago which explains what are the slow and fast lanes but that is where the issue is I believe. You have several options to decouple the fast lane but the only one that can work in your case is an auxiliary \$V_{cc}\$ coming from an auxiliary winding. Considering the ratio of 34/4 which is 8.5 you may have to think of a clever solution to produce this extra stable voltage. I have used one of my free SIMPLIS platforms to simulate a 34-V forward converter whose output can be folded back to 4 V:
The calculations of the type 2 featuring a disconnected fast lane are given in the right-side sheet and they automate the process in SIMPLIS. I have shot for a 1-kHz crossover and, as expected, the power stage response does not change significantly in relationship with the output voltage setpoint. I have a decent phase margin of nearly 60° in both cases, 34 V and 4 V output. The low-side resistance of the divider is changed for doing so: don't touch the upper resistance as it would affect the compensator response while the low-side does not (only the dc setpoint changes). When you follow these recommendations, the response at 34 V out is ok:
and so it is when \$V_{out}\$ is set to 4 V:
Finally, the transient response shows a good stability at both output voltages: