Disclaimer: I'm answering your questions as I understand them. You will have to come back to me if I misunderstood you. First some background:
- Differential signaling and ground reference
In differential signaling, the signal is not referenced to ground, so in theory you don't need a ground connection between transmitter and receiver. However, you need a way to ensure that the receiver's common mode voltage range is not being violated. Since the common mode voltage on the receiver side is measured with respect to the receiver's own ground, this amounts to controlling/limiting the voltage difference between the transmitter's ground and the receiver's ground. It is not necessary to connect them directly, but there needs to be some sort of arrangement that limits the voltage difference in practice. The simplest such arrangement, of course, would be an explicit ground connection. This could be a metallic shield in the CAN cable.
Note that even with isolation, this situation isn't different in principle. However, the common mode voltage range is usually orders of magnitude greater than without isolation.
- Isolated CAN transceivers
Note that isolated CAN transceivers don't have the isolation barrier directly at the cable. There is some amount of electronics on the cable side (the actual transceiver), and this must be supplied with power. For this transceiver, the discussion above applies unchanged. The transceiver has its own ground, and its common mode voltage limits with respect to this ground.
The isolated CAN transceiver includes another chunk of electronics on the other side of the isolation barrier, and this part has its own ground. Its voltage can differ from the transceiver's ground by a substantial amount.
Your example of the ADM3053 also includes an isolated DC/DC converter to supply the transceiver with power. This allows you to have the electronics of your CAN node work from an arbitrary power source with its own ground level, but you still need to treat the transceiver ground (called GND2 in the ADM3053) in the same way as described above in 1.
- Floating ground
There is no such thing as a floating ground. In fact, there is no such thing as ground. There are only different signals we call "ground" (often somewhat arbitrarily), but this doesn't change their physical nature. A floating ground typically means that there is a signal we call "ground", which is coupled to another signal that we also call "ground", through a fairly high impedance (often this is predominantly capacitive).
In the context of a differential CAN bus, you invariably have - at least conceptually - a ground signal the bus is referenced to. It might be a chassis or metal structure everything is mounted on or in. It might be the cable shield. Regardless of whether you use non-isolated or isolated transceivers, you need to ensure that their common mode voltage restrictions are obeyed with respect to this ground signal. As the bus is differential, some amount of ground level variation is tolerable between the different points on this distributed ground structure, without impairing the communication.
- Isolated nodes
If you want to connect a node to the CAN bus, whose own ground can not be guaranteed to obey the rules laid out above, you need an isolated transceiver, which permits you to keep the ground for the node separate from the ground for the bus. This doesn't change the situation for the bus, but it does change the situation for the node. The node itself now has to deal with two different grounds, with potentially a large voltage between them. That's what the isolated transceivers are made to help with.
With this background, let me try to answer your 3 questions:
- "How does isolated DC2DC behave? What prevents the GND of device B and device A from being slightly different in magnitude?"
You need to be clear here which ground you mean. I assume you mean the ground of the node in question, and not the bus ground.
Due to the fact that you have an isolated transceiver, which also includes an isolated DC/DC converter to supply the transceiver circuit, you have three different grounds, which can all have a substantial voltage difference between them: The ground of node A, the ground of node B and the bus ground. Nothing prevents them to float against each other, and no problem results from this, as long as the breakdown voltage limits of the isolation barriers are respected.
- "What is the most suitable way to connect the GND to ensure that the reference is the same? Otherwise nothing prevents the two devices to have very large GND difference and therefore to be be in the common-mode range."
This depends on your concrete situation. In a car, for example, it will usually be adequate to use the car's chassis as the ground connection for the bus. The ground of all transceivers, isolated or not, should be connected to it. In isolated transceivers, however, the node ground (GND1 of the ADM3053) would typically not be connected to that.
In another situation, you may want to use a shielded bus cable, and use the shield as the bus ground connection.
- "If a network has an ADM3053 (isolated) and a TCAN1044V-Q1 (non isolated), TCAN1042HGV-Q1, TJA1441 (non isolated) what is the best practice to connect all the GND references? Few devices have an isolated transceiver, few have isolated DC2DC, and some have neither, i.e are non-isolated."
It again depends on the situation, similar to the previous question. The rule is that the grounds of the transceivers get connected to bus ground in one way or another. If the local ground of a node can't be connected to that, you need to use an isolated transceiver, in the way described above. If the transceiver doesn't include a DC/DC converter, you need to worry where the power for the actual transceiver circuit is coming from. Perhaps your bus cabling includes a wire for supplying power to the transceivers. In that case you can use this, instead of needing an isolated DC/DC converter.
TLDR; Connect all transceiver grounds to "bus ground". When necessary, keep a node's ground separate from that, through the usage of an isolated transceiver. But in general, be conscious of the different grounds, their function/purpose, and the currents flowing in them, even in case of a fault.
