When you set out to make something that can support differential or single ended connections from the encoder why not try what I have done on a number of projects.
Before delving into that let me clarify for you that differential encoders do not output voltage swings that go above and below GND. A single A+ or B+ or A- or B- will always swing from GND to some positive voltage (very typically 5V). The differential aspect comes onto play simply because the A- and B- pins are simply logically inverted from their A+ and B+ counterparts.
Also note that differential receivers are designed to pull the difference out from between a pair of signals that are nominally logical inversions of each other. The typical differential receiver will be capable of detecting from a signal pair that has reduced amplitude, even down to less than half a volt, due attenuation through a very long cable. The receivers also can accommodate a common mode voltage range that extends below GND so that they can detect in situations where there can be a number of volts of GND level shift between different subsystems. (Even without signal level attenuation the receivers in a differential system are also capable of rejecting high noise levels on each signal of a pair).
In my designs I bring in from a connector all of A+, A-, B+ and B-. The A+ and B+ inputs go directly to the + inputs of a AM26LV32 quad differential line receiver. (Note that some designs will also require the Z or Index channel as well). The A- and B- inputs go to a dual 2->1 analog mux chip (I use the TS5A22362 part). The outputs of the switch chip then go to the - inputs of the AM25LV32.
The other side inputs of the analog mux chip come from a TLV431 shunt reference chip setup to produce a voltage level of ~2.5V. A control line from my MCU is used to switch the analog mux for either the single ended mode or the differential mode. In the single ended mode the AM26LV32 is seeing its + input swing approx 0 to 5V and the - input is set at a mid point threshold from the TLV431. (Note that this really assumes that the encoder is not 100's of feet away and as such does not expect there to be a huge GND shift between the encoder and the receiver. With a 2.5V reference and 0-5V swing signals a GND voltage difference between encoder and receiver of a up to a volt can still be easily accommodated for single ended operation).
The AM26LV32 is a great part because it can take the 0 to 5V inputs and run with a VCC of 3.3V. This permits automatic level conversion of the outputs to 3.3V logic levels for connection to the MCU (or as in my typical case an FPGA).
I also anticipate that some encoders that operate in single ended mode may be connected in an open drain / open collector configuration with output pullups to a higher voltage such as +12V. I take care of that in my design by running the encoder signals through a 2.2K resistor before presenting them to the above described circuitry. The downwind side of the 2.2K resistor is also connected off to an ESD / voltage clamp diode chip that is connected into the +5V supply. I use the DVIULC6-4SC6 for these clamps. The beauty of this setup is that it allows for encoders that may be outputting voltages in the range of 0 to as high as 15V or 16V but works just as well for 0-5V encoders.
If the reference was changed to something more like 1.7V to 2V then the circuit would even work nicely for single ended encoders that operated at 0->3.3V voltage swings.