0
\$\begingroup\$

I am trying to send data (not Ethernet data, but sFPDP data) from a PCIe device to a Infiniband device. I know Infiniband sends its data serially vs parallel for PCIe.

If I want to send data from my PCIe device to my Infiniband devices, would it be as simple as just getting a mellanox switch and just routing all the PCIe ports to the one Infiniband port? I know they both use QSFP cages. Seems like I would just need to make sure the QSFP cables are HDR or FDR compliant?

\$\endgroup\$

1 Answer 1

2
\$\begingroup\$

Infiniband and PCIe are related but very different. Even the link training state machine (LTSSM for PCIe) is different as is the method of determining whether a link partner is present.

They are both based on serial differential pairs that may be used as lanes on a single logical port (although the number of possible lane counts lanes is a bit different).

The packet structure for the two is not the same and the flow control although of the same principle is also different.

Even the type of interconnect is different (PCIe is a root complex with endpoints and Infiniband is a fabric).

So Infiniband and PCIe differ significantly both electrically and logically.

The bottom line is that you cannot just hook one up to the other; you will need a target channel adapter (or host channel adapter which is more capable) on the Infiniband side with a processor that can then forward the data to the PCIe device.

Some devices from Mellanox cater for multiple protocols on different ports; whether there are any products that implement such a bridge is something I do not know.

You may find this comparison useful.

\$\endgroup\$
1
  • \$\begingroup\$ NVidia (Mellanox) does sell ConnectX chipsets that allow you to connect your board level PCIe to Infiniband. They are complex, expensive and NVidia wants to be involved in the design reviews due to the high speed signal layout. \$\endgroup\$
    – edt11x
    Commented May 29, 2022 at 15:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.