ItIt's simple. Yes, you need a gnd connection so you can ensure common mode is not floating away out of the receiver ratings. The gnd connection may not necessarily be a good one, short or whatever, but it has to be there.
The only exclusion is when you have some other weird way to ensure that it does not float, like maybe connecting gnd on one side to an isolated vcc on the other side. But it's weird and takes very careful analysis of both sides.
One more thing. There is a standard way to handle encoders: provide power (5V and GND) in the same cable where the communication goes. Anyway on the encoder side there is nothing electric. A kind of a dead end.