On my prototype PCB, I have a microcontroller with reserved pins for UART communication with a debug terminal. I'm curious from a theory perspective about why my terminal works fine whether or not I connect ground to my USB UART adapter. I come from a computer science background, so assume my EE fundamentals are lacking.
Check whether your connector's shields (E.G. MicroUSB) are not doing the job of linking the grounds of the systems. Depending on how the USB to UART converter board you're using was built, the GND might be connected through the connector shielding and along the cable shielding. I'd start by measuring continuity between system's grounds with everything powered off.