I'm guessing this sparkfun level converter is what you are using as it looks like what you have in the pictures. Edit: It's not the sparkfun product, but looks like exactly the same topology. No datasheet or schematic is provided for the Anmbest product so I can't verify part numbers.
Take a look at the schematic and you will find it illuminating. The level converter is just a MOSFET and some pullups. The "direction" of the bi-directional level converter assumes one side at a time being connected to a high impedance, and the other being connected to a low impedance driven connection.
When you leave HV1 unconnected, HV1 is pulled high (5.0V) by the resistor connected to HV, and there is nothing to pull LV1 low, so LV1 is pulled high (3.3V) by its resistor connected to LV, and you read the high output.
If LV1 is the output, the only way to get a low signal is to drive HV1 low, which will pull LV1 down through the MOSFET body diode, which will then turn on the MOSFET since its source is connected to LV1 and its gate to LV (3.3V).