Why doesn't PCIe and similar signaling systems use full-duplex links?

Question

I'd like to get more PCIe bandwidth for GPU compute applications. It occurred to me that PCIe bidirectional links are really dual simplex (a pair of unidirectional links). That means if there's no data to transmit in 1 direction, 1/2 the links are idle!

So, my question is obviously, why not allow the links to be bidirectional (half-duplex)?

I can't think of any other networking physical layer that's bidirectional, except I2C and Wifi, which I doubt can reach GhZ speeds. Is high speed incompatible with bidirectionality?

Update: I did think of a high speed bidirectional interface: a memory bus. XDR memory uses differential signaling and claims 20 Gbits/s per pin! Why does it work for memory but not for PCIe?

Answer 1

I think you are assuming here that the number of physical lanes is the bottleneck, and that if you only had more bandwidth then the logic inside the GPU would no longer be held back. I don't think that is really the case. To give you more bandwidth in one direction you need more area, power and resources dedicated to say receivers. Then your whole pipeline has to be designed to support this new double dataspeed. Not to mention a high speed serdes is a complex beast, so now you'd be making each piece even harder by trying to support both RX and TX.

When designers want to double bandwidth it's simple to just add lanes, just step and repeat. When PCI-SIG wants to increase the max speed for the next generation it's easier to just add features and increase frequency.

PCIe in particular is based on the concept of each lane pair being bi-directional. From the time it starts up and negotiates speed and TX/RX parameters, to later on for flow control. If you decided to take away one TX and change it to an RX, now another TX in a separate part of the chip would have to take over it's duties and it all gets pretty messy pretty fast.

Technically though if someone wanted to design a HS link system that allowed you to change your TX to RX for extra receive bandwidth it could be done. But it really would only make sense if pins were expensive because why add all the complexity to your system and design to support this when you could just add a few pins.

Answer 2

I feel bad about answering myself, but unlike most other people who think it can't be done efficiently, I think the answer is more qualified.

I agree that it can't be done for PCIe. The main assumption is that the communication is peer to peer. Then, it would be pretty difficult like SomeHardwareGuy claimed because if you're trying to send and there are no available TXs, you would have to buffer your packet.

But bidirectional communication at GHz speeds is possible on a memory bus. The key difference has to be that only the memory controller initiates communication, so it's known precisely in which clock cycles who should transmit and who should receive (Wow, so those memory timings (e.g. tCAS) aren't just for bragging - they're actually for synchronizing the memory controller with the memory). That means no need to buffer packets and no need to negotiate whether a links is to be used for RX or TX.

Answer 3

Is high speed incompatible with bidirectionality?

Yes, because of turnaround time. It takes roughly 1ns for a signal to travel 30cm, plus delay through the TX and RX at each end. So it will take at least several nanoseconds, possibly hundreds of nanoseconds, to reverse direction.

Then you have to work out how to control the switchover, which makes the signalling more complex and imposes a latency of its own. Note that PCIe lanes may not have exactly identical propagation delay, so signals that set off at the same time may arrive at different times.