0
\$\begingroup\$

According to Wikipedia

PCI Express 3.0's 8 GT/s bit rate effectively delivers 985 MB/s per lane

985MBps * 8 = 7880MBps (63Gbps).

If I am using a 10Gbps Ethernet NIC with PCIe 3.0 8x interface and I am sending or receiving data at full 10Gbps line rate, I assume that my data between the OS and network card is being multiplexed over multiple PCIe channels in the motherboard slot my card is in.

Assuming that is correct, how does PCIe maintain packet ordering when multiplexing over multiple channels or is there no data ordering support? How am I getting 10Gbps of throughput with 8Gbps per channel?

|improve this question
\$\endgroup\$
  • \$\begingroup\$ You won't get 10gbit throughput. \$\endgroup\$ – PlasmaHH May 16 '17 at 9:22
  • \$\begingroup\$ So how do 10Gbps cards get 10Gbps? \$\endgroup\$ – jwbensley May 16 '17 at 9:30
  • \$\begingroup\$ Most don't; those that do have a higher speed host interfaces. The majority of OS and application configurations won't be able to run with that speed anyways. \$\endgroup\$ – PlasmaHH May 16 '17 at 9:32
  • \$\begingroup\$ Oh, I just now saw that you refer to 8x interfaces... well, MB is bytes... \$\endgroup\$ – PlasmaHH May 16 '17 at 9:34
  • \$\begingroup\$ I just put 8x as an example, perhaps I should remove that from the question or change it to 16x? We have 10Gbps cards that run at line rate 10Gbps so somehow, it happens (I don't know if they are 8x or 16x channels though). \$\endgroup\$ – jwbensley May 16 '17 at 9:35
1
\$\begingroup\$

The card is effectively a Serializer/Deserializer (SerDes for short). One possible implementation is there's a clock for the parallel bus running at PCIe 3.0 speed, which latches the incoming 8 bits of data. Then internally there is a shift register running at a faster clock (10Gb ethernet speeds) which just shifts that out. The process can be performed in reverse to convert ethernet data into PCIe signals.

There is a strong correlation between which PCIe pin corresponds to which bit of the parallel buffer, which is what guarantees the correct ordering in both transmit and receive mode (ex.: pin 0 is always bit 0, pin 1 is always bit 1, etc.). Parallelism is exposed at the bit level.

There is more complications when dealing with the actual PCIe/10G ethernet protocols, (10Gb ethernet uses a 64b/66b encoding), but this basic picture mostly sums up the broad idea.

|improve this answer
\$\endgroup\$
  • \$\begingroup\$ PCIe doesn't split across lanes at the bit level at all. It splits at the byte level. For Gen1/2 each block will be a single byte that is 10/8b encoded. The bytes split over the lanes so lane 0 gets byte 0, lane 1 gets byte 1, and so on. For Gen3, each block is again split into bytes but each block contains 16 bytes (plus 2 bits of header) and uses 132/130b encoding - take 4 lanes for example and lane 0 will get bytes 0,4,8,...,60 in its block, lane 1 gets 1,5,9,...,61, and so on. \$\endgroup\$ – Tom Carpenter Sep 20 '17 at 22:00
  • \$\begingroup\$ Furthermore each lane is handled completely separately at the physical layer such that lane to lane skew doesn't cause problems. Data is realigned at the other end by sending special control blocks that indicate starts/ends of data chunks, and also skip (no-operation) blocks which get inserted to realign FIFOs in the event of skew. \$\endgroup\$ – Tom Carpenter Sep 20 '17 at 22:02
  • \$\begingroup\$ At either end the data gets aggregated and aligned so that the lanes become transparent to higher up levels of interface handling. The separation between layers is how PCIe can handle working with fewer than native lanes (e.g. x16 card in x4 slot) without massive headaches in the implementation. \$\endgroup\$ – Tom Carpenter Sep 20 '17 at 22:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.