4
\$\begingroup\$

I'd like to get more PCIe bandwidth for GPU compute applications. It occurred to me that PCIe bidirectional links are really dual simplex (a pair of unidirectional links). That means if there's no data to transmit in 1 direction, 1/2 the links are idle!

So, my question is obviously, why not allow the links to be bidirectional (half-duplex)?

I can't think of any other networking physical layer that's bidirectional, except I2C and Wifi, which I doubt can reach GhZ speeds. Is high speed incompatible with bidirectionality?

Update: I did think of a high speed bidirectional interface: a memory bus. XDR memory uses differential signaling and claims 20 Gbits/s per pin! Why does it work for memory but not for PCIe?

\$\endgroup\$
  • 2
    \$\begingroup\$ Think of the time taken to reverse communication direction : tristate outputs, transmit sync patterns, resynchronise inputs... yes, high speed is fairly incompatible with bidirectionality. \$\endgroup\$ – Brian Drummond Nov 17 '14 at 22:21
  • \$\begingroup\$ What is "GhZ"?? \$\endgroup\$ – Andy aka Nov 17 '14 at 22:29
  • 1
    \$\begingroup\$ "I can't think of any other networking physical layer that's bidirectional" --- Have a look at how Gigabit Ethernet is implemented at the chip level. GigE chips see the interface as simplex connections; external magnetics are used to create duplex connections on the cable. \$\endgroup\$ – The Photon Nov 17 '14 at 22:56
  • 1
    \$\begingroup\$ Good point. But half-duplex is no longer an option for 10GigE. It seems that's because collisions would make it very unusable (very high bandwidth * delay product means you'll throw away a lot of work). \$\endgroup\$ – Yale Zhang Nov 18 '14 at 21:59
5
\$\begingroup\$

I think you are assuming here that the number of physical lanes is the bottleneck, and that if you only had more bandwidth then the logic inside the GPU would no longer be held back. I don't think that is really the case. To give you more bandwidth in one direction you need more area, power and resources dedicated to say receivers. Then your whole pipeline has to be designed to support this new double dataspeed. Not to mention a high speed serdes is a complex beast, so now you'd be making each piece even harder by trying to support both RX and TX.

When designers want to double bandwidth it's simple to just add lanes, just step and repeat. When PCI-SIG wants to increase the max speed for the next generation it's easier to just add features and increase frequency.

PCIe in particular is based on the concept of each lane pair being bi-directional. From the time it starts up and negotiates speed and TX/RX parameters, to later on for flow control. If you decided to take away one TX and change it to an RX, now another TX in a separate part of the chip would have to take over it's duties and it all gets pretty messy pretty fast.

Technically though if someone wanted to design a HS link system that allowed you to change your TX to RX for extra receive bandwidth it could be done. But it really would only make sense if pins were expensive because why add all the complexity to your system and design to support this when you could just add a few pins.

\$\endgroup\$
1
\$\begingroup\$

Is high speed incompatible with bidirectionality?

Yes, because of turnaround time. It takes roughly 1ns for a signal to travel 30cm, plus delay through the TX and RX at each end. So it will take at least several nanoseconds, possibly hundreds of nanoseconds, to reverse direction.

Then you have to work out how to control the switchover, which makes the signalling more complex and imposes a latency of its own. Note that PCIe lanes may not have exactly identical propagation delay, so signals that set off at the same time may arrive at different times.

\$\endgroup\$
  • 1
    \$\begingroup\$ I would think turnaround-time issues would exist any time one device can hear transmissions from more than one other, but in cases where one device wishes to communicate with one other, I would think that even simultaneous communication over a single twisted pair should be workable (Plain Old Telephone Service lines have been doing it for ages). POTS lines are slow, but I don't know of any reason the same principle couldn't be used at higher speeds. Not sure it would be worthwhile, but it would certainly seem doable. \$\endgroup\$ – supercat Jan 14 '15 at 20:29
  • 1
    \$\begingroup\$ @supercat this is done with Gigabit ethernet and 10GBASE-T (simultaneous in both directions on four pairs). However, these both use heavily encoded multilevel signaling (PAM-5 and PAM-16, respectively) which require DACs and ADCs to transmit and receive, quite a bit of DSP circuitry to implement echo cancellation and equalization, and add quite a bit of latency. 10GBASE-T also uses FEC. So yes, it may well be possible, but it would come at a significant increase in complexity and therefore cost. For reference 10GBASE-T is 4 lanes at 2.5 Gbps, equivalent to 4 lanes of PCIe gen 1. \$\endgroup\$ – alex.forencich Mar 6 '16 at 4:03
1
\$\begingroup\$

I feel bad about answering myself, but unlike most other people who think it can't be done efficiently, I think the answer is more qualified.

I agree that it can't be done for PCIe. The main assumption is that the communication is peer to peer. Then, it would be pretty difficult like SomeHardwareGuy claimed because if you're trying to send and there are no available TXs, you would have to buffer your packet.

But bidirectional communication at GHz speeds is possible on a memory bus. The key difference has to be that only the memory controller initiates communication, so it's known precisely in which clock cycles who should transmit and who should receive (Wow, so those memory timings (e.g. tCAS) aren't just for bragging - they're actually for synchronizing the memory controller with the memory). That means no need to buffer packets and no need to negotiate whether a links is to be used for RX or TX.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.