0
\$\begingroup\$

AR 1215986 was mentioned on page 7 of PG344, Versal Adaptive SoC DMA and Bridge Subsystem for PCI Express Product Guide.

In this AR, the author mentioned several components, namely:

  • PCIe PHY
  • GT QUAD
  • PCIe MAC
  • Wrapper

I am trying to understand the "protocol stack order" of these components.

  • The wrapper is on the top of the stack I believe, be it a DMA or bridge.
  • PCIe MAC: I guess this is what usually called "Controller". If this is the case, it should contain TLP/DL/MAC layers. I don't know why it's called "PCIe MAC" in the AR (i.e., without mentioning the upper layer such as TLP/DL).
  • GT QUAD and PHY: I am quite confused about the relationship between this two. I had an impression that GT is serdes, which has PCS/PMA layers, which is pretty much a PHY by itself. Why there are separetly listed here?

I create a test BD in Vivado (2023.2), with the following block diagram (after removing the extra "support" hierachy, attempting to have a better view of all PCIe related components).

enter image description here

The picture suggests a stack order (in the sense that "left to right" represents "top to bottom"), such that GT is the bottom component. But then the pcie_mgt signals from "pcie_phy" module confuses me: why the external connection pcie_mgt is coming out from a middle layer in the stack (pcie_phy), instead of from the bottom layer (gt_quad_0)?

\$\endgroup\$
1
  • \$\begingroup\$ Looking at the signals of each ip blocks, it seems the traffic path is (taking tx for example): [pcie/phy_mac_tx] -> [phy/phy_mac_tx -> phy/GT_TX0] -> [quad/TX0_GT_IP_interface -> quad/GT_Serial] -> [phy/GT_Serial -> phy/pcie_mgt] -> external_pins. \$\endgroup\$
    – bruin
    Commented Jan 22 at 6:17

2 Answers 2

1
\$\begingroup\$

I might be able to help as I have been here before when I had to build the entire PCIe ip all by myself (although it took me over a year).

You are looking at 4 things here...

  • GT Transceiver
  • Phy for PCIe
  • Integrated Block for PCIe Express
  • Queue DMA subsystem for PCIe

1. GT Transceiver (Quad): This is an IP that is configuring the High Speed Serial IOs for use according to PCIe standard data rates. Mainly, three things happen here:

  • Clock Data Recovery/Merging for Rx/Tx.
  • encoding/decoding 8b/10b or 64b/66b as per PCIe Gen.
  • Dealing with Control Characters. In PCIe documentation there is no such term as quad or transceivers but instead they are described as part of physical Layer. There is a wrapper attached with Quad to make the standard interface with the Quad and its called PIPE Interface.

2. Phy for PCIe: consists of two parts. One part is what is discussed in above section. The second part is where following is implemented in phy IP that you are seeing:

  • LTSSM: (Link Training and Status State machine)
  • Byte Alignment and Skew Elimination
  • Scrambling and Descrambling
  • Controller for the GT Transceiver

Together, the GT Transceiver and Phy for PCIe, they make the Physical Layer of the PCI Express, as mentioned in the PCIe Base Specification Documents.

3. Integrated Block for PCIe Express: This is the complete PCIe IP that is mentioned in PCIe standard documents. It contains three layers of PCIe:

  1. Physical Layer
  2. Data Link Layer (DLL)
  3. Transport Layer (TLP)

So in your block design, Integrated Block for PCIe Express, via PIPE interface, is instantiating the physical Layer IP i.e. Phy for PCIe which is further controlling the GT Transceivers. The other two layers (DLL and TLP) are also implemented in the Integrated Block for PCIe Express but their hierarchy might not be simply visible because it is implementation specific. Also, there are no specific interface between the DLL and TLP so there implementation is kind if merged for resource sharing etc. The output/input of this Integrated Block for PCIe Express IP is a set of streaming ports that provide/accept PCIe standard packets to/from the user.

Along with that there is also a configuration memory in the Integrated Block for PCIe Express where all the PCIe specific registers resides. lspci command on Linux can be used to view the register contents of a specific PCIe Device.

Data Link Layer (DLL), Transport Layer (TLP), Phy for PCIe (as discussed in section 2.) and Configuration Memory, together make the MAC of PCIe. You can find more details about MAC, PHY and PIPE interface here: https://www.mindshare.com/files/resources/mindshare_intro_to_pipe_spec.pdf

4. Queue DMA subsystem for PCIe: This is the application layer of the PCIe. It implementation is user specific. Queue DMA subsystem for PCIe is one example of application layer, but user may implement a new application layer as per his requirement. There are also a lot of open source implementations available for the application layers. This layer is what makes the actual use of PCIe IP by sending and receiving data.

Here is a picture to elaborate more about the above discussion:

PCIe Blocks as per Xilinx Implementation

These are just the basics about the PCIe that I can summarize here. I will be glad to help if you want to discuss further about the PCIe.

Hope that helps.

\$\endgroup\$
6
  • \$\begingroup\$ Thanks for your elaborated answer, @Im Groot. As for my original question (regarding the placement of pcie_mgt signals), I am kind of "explained to myself", that the GT Quad is really part of the the PHY, and they are just seperated in the block diagram shown in Vivado. Btw, you mentioned 64/66b in GT section, I guess you meant 128/130 for PCIe. Another point is about locaiton of LTSSM, you mentioned that it's implemented in PHY IP, but from what I understand for now, LTSSM is not in Xilinx PHY IP but in MAC layer (i.e., above the PIPE interface). \$\endgroup\$
    – bruin
    Commented Feb 5 at 23:04
  • \$\begingroup\$ A related question is about MAC layer: in PCIe, it seems that MAC is regarded part of the PHY (i.e., in the part of PHY above PIPE); whereas in ethernet, MAC is regarded as part of Data Link layer. Why such differences? \$\endgroup\$
    – bruin
    Commented Feb 5 at 23:08
  • \$\begingroup\$ Another minor question is about the naming of pcie_mgt: what mgt stands for? I guess it's not management since it's usually abbreviated as mgmt, so mgt may mean Multi-Gigabit Transceiver? Thanks again for your answers. \$\endgroup\$
    – bruin
    Commented Feb 5 at 23:11
  • \$\begingroup\$ I have updated my answer, I made a bit mistake in Describing about PIPE and MAC definition. I sort of confused it with an implementation of mine. \$\endgroup\$
    – Im Groot
    Commented Feb 6 at 14:04
  • 1
    \$\begingroup\$ Thanks. Yes, I saw the same diagram from PG345, and that's the diagram leads me to make a conclusion that GT is actually part of the PHY, albeit it's separated out in vivado block design for versal devices. \$\endgroup\$
    – bruin
    Commented Feb 7 at 0:55
1
\$\begingroup\$

My interpretation would be:

  • the quad is the four SERDES channels that form a group. Nothing PCIe specific goes on here, and the channels are as independent as they can be (IIRC they share the PLL for the TX clock generation, and the per-channel PLLs are used for CDR).
  • the PCIe PHY performs byte and word alignment during link training, and is responsible for splitting data across lanes later. This requires knowledge of the PCIe protocol, such as the training sequences, so it cannot be part of the protocol independent layer below.
  • the PCIe MAC handles packet generation and processing.

These are separate entities, because the higher level functions can be implemented either as hard IP, or as regular logic, if some deviation from the protocol is required.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.