I think I can "piece together" your problem now.
I am using the above system
In your update, you seem to refer to the LTC2863 pinout. Therefore much of the confusion was that you originally mentioned in a (broken) link, that you were using the LTC2862 not the LTC2863. :-(
Assuming you really did mean LTC2863, then:
I am using the LTC2863 which has only 1 diff tranciever.
Not exactly. The LTC2863 has 1 differential transmitter (driver) output and 1 differential receiver input. It has no combined transceiver connection. (The LTC2862 has a single transceiver connection pair.) The fact that the LTC2863 has separate transmitter and receiver connections does allow it to have a loopback attached to it.
I am connecting A* to Y* and B* to Z*. Should this loop back work ?
Yes, that should work correctly. (FYI, you don't need the "*" characters - that is just shown on some of their diagrams, because those pins have 15 kV HBM ESD protection.)
Here is the explanation about what those pins are:
You would be connecting pin
Y to pin
A (non-inverting driver to non-inverting receiver) and pin
Z to pin
B (inverting driver to inverting receiver). Thereby correctly connecting the inverting signals together and the non-inverting signals together.
Although it is a slightly unusual example circuit, this diagram from the datasheet confirms that those are the correct connections (driver pin
Y connected to receiver pin
A and driver pin
Z connected to receiver pin