2
\$\begingroup\$

could someone please do for me the calculation of total throughput of PCI-E? The best material I could find was a Xilinx pdf that mentioned 2.5Gbps as reference value and somewhat awkward formula that didn't utilize clock speed. You see, I had no problem just memorizing this thing until someone said 100Mhz and now I want to know how does 100Mhz become 2.5 Gbps.

EDIT

Both answers are very good. Thank you. Unfortunately, I can only mark one.

\$\endgroup\$
2
\$\begingroup\$

First of all, the 100MHz you refer to is the reference clock frequency. The reference clock is either shared between the motherboard and devices, or created from a local oscillator. This is not part of the throughput calculation.

To get the transceiver clock frequencies (the frequency of the high speed TX and RX lines), a Phase-locked loop (PLL) device is used to step this up the reference clock frequency to a higher value. The clock rates are 1.25GHz (2.5 Giga-transfers per second (GTps)) for PCIe Gen 1, 2.5GHz (5GTps) for PCIe Gen 2, or 4GHz (8GTps) for PCIe Gen 3.

To work out the throughput, you need to know a few extra things.

  • First of all, the transceiver data rate is not the same as the usable data rate. PCIe transceivers use an encoding scheme for the data to ensure there is no DC component in the data signals amongst other things. For Gen 1 and 2, an 8:10b scheme is used (10 bits sent for every 8 bits of data), and 128:130b for Gen 3.

  • Secondly your data is not sent directly, but rather as packets (like network traffic). These Transaction-Layer-Packets (TLPs) have a 16-byte header for each chunk of data. The data chunks can range from 1-byte to 4095-bytes in size. Typically the payload size is no more than 256-bytes in PCs.

  • Thirdly, there is other data sent across the link at lower parts of the protocol (i.e. Data-Link Layer), such as realignment packets, flow control (data credits), CRC checksums, and interrupt packets.

Putting that together, the calculation becomes:

$$ \mathrm{Lane Data Rate}\times\mathrm{LaneCount}\times\mathrm{Encoding Scheme}\times\mathrm{Efficiency} $$

For Gen 1/2, \$\mathrm{Encoding Scheme}\$ is \$0.8 (\frac{8}{10})\$, and for Gen 3, \$\mathrm{Encoding Scheme}\$ is \$0.985 (\frac{128}{130})\$. This gives the ideal through-put of TLPs (including header) of 2Gbps per lane, 4Gbps per lane, and 7.88Gbps per lane for Gen 1, 2 and 3 respectively.

As to what the actual data through-put is, that depends on the efficiency. Assuming that you manage to sent back to back 256byte payload TLPs, your \$\mathrm{Efficency}\$ would be \$0.94 (\frac{256}{256+16})\$.

However in practice, there will be other devices on the bus sending data, and the consumer (device data is being sent too) may not be able to handle that rate. Additionally there are the other packets used in lower levels of the protocol for aligning the transceivers, and flow control. These will all reduce your overall efficiency.

In a system based on a high-end Stratix V FPGA transferring data from the FPGA to DDR4 memory in an Intel i7 PC, I've been able to achieve a sustained throughputs of around 57Gbps on a Gen3x8 interface. In terms of \$\mathrm{Efficency}\$, that equates to around \$0.9 (90\%)\$ over some very large (multi-gigabyte) transfers, or 7Gbps per Gen3 lane.

\$\endgroup\$
  • \$\begingroup\$ There is more overhead than 4 bytes per TLP, don't forget about the link layer \$\endgroup\$ – alex.forencich Apr 16 '18 at 17:44
  • \$\begingroup\$ And the TLP headers are 3 or 4 dwords, so 12 or 16 bytes \$\endgroup\$ – alex.forencich Apr 16 '18 at 17:47
  • \$\begingroup\$ @alex.forencich Aye, 4 dwords, had a brain fart. I realise they can also be 3DW for 32-bit headers, but I'm trying to simplify. I'm also aware of the link layer stuff, that's what I refer to in the third bullet point and the second to last packet. \$\endgroup\$ – Tom Carpenter Apr 16 '18 at 17:57
  • \$\begingroup\$ Well, there's also the sequence number and LCRC attached to each TLP. Looking at the spec now to see how many bytes that is. \$\endgroup\$ – alex.forencich Apr 16 '18 at 17:58
  • \$\begingroup\$ The gen 3 spec lists a figure of 28 symbols overhead per TLP for flow control calculations. Apparently this includes TLP header, link layer fields, and control characters. Not sure what the exact breakdown is. \$\endgroup\$ – alex.forencich Apr 16 '18 at 18:00
3
\$\begingroup\$

Simple calculation would just be number of lanes times the lane rate. The lane rates are 2 Gbps for PCIe gen 1 (2.5 Gbps raw), 4 Gbps for PCIe gen 2 (5 Gbps raw), and 7.877 Gbps for PCIe gen 3 (8 Gbps raw). The lane rates are lower than the raw serializer rates due to encoding overhead (8b/10b or 128b/130b). The number of lanes is usually going to be 1, 2, 4, 8, or 16. So, a gen 3 x16 card would give you 7.877*16 = 126 Gbps. I think there may be some other lane counts possible, but they are not very common. If you want to do a more thorough calculation, then you have to look at PCIe transaction layer packet (TLP) overhead for reasonable packet sizes, which could be system and application dependent.

The 100 MHz is just a reference clock that's distributed to components in the system. Each card will feed that reference clock through a PLL to generate the high frequency clocks required to operate at the desired link rate.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.