RF port and LO port are inter-changeable. If you imagine that LO-to-IF is balanced, then RF-to-IF is balanced as well.
But any output at the IF port does cause unbalance. In normal mixer use, both RF signal level and IF signal levels are small...much smaller than LO signal level.
Any signal developed at the IF port causes unbalance that will transfer a fraction of that large LO power to the RF port. That back-flowing LO power can do no good in the common mixer scenario.
The unbalance goes both ways: some RF port signal is transferred to LO port, but that scenario isn't usually a worry (because RF port power is tiny compared to LO port power).
This DBM circuit is sometimes used as a variable attenuator. In this case, the unbalance is put to good use: a DC current applied to the IF port controls signal transfer between RF port and LO port.
The other application is a modulator, where the baseband modulation signal is applied to the IF port. The RF port becomes the modulator output. Were it not for the unbalance @ IF port, this wouldn't be possible.
My use of the term "unbalance" above is the process by which an RF port signal provides desired IF port output. A mixer-maker defines unbalance differently: it is what destroys symmetry. It is caused by diode mis-match, transformer irregularity, stray capacitance and inductance.
Since the LO port has the largest signal by far, a mixer-maker may concentrate on measuring LO port -to- RF port isolation, and LO port -to- IF port isolation. The LO port is driven with a large signal, at the proper impedance (usually 50 ohms). The RF port is terminated with 50 ohms, and IF port with 50 ohms passively. Perfect symmetry would transfer no power from LO to RF, and would transfer no power from LO to IF.
But be aware that this test setup meant to measure port-to-port isolation isn't doing any useful mixing.