1
\$\begingroup\$

I want to construct a matrix of smart boards that receive ethernet packets, decode them, and place the decoded results onto a memory "matrix" for other boards to process and re-transmit. Given the available technologies, I suspect PCIe is the way to go. I have a couple of questions for you all:

1) Who controls the PCIe bus? I suspect its the (PC) motherboard processor at a high level, and the PCIe controller at a lower level, but its not clear.

2) Is PCIe designed for lots of small transfers, or a fewer number or larger transfers? I suspect the latter, as the former would kill the motherboard cpu...

What do you think?

\$\endgroup\$
  • 1
    \$\begingroup\$ Is this meant to be used inside a PC or as a standalone system? What kind of bandwidth and latency do you need? In what form is the data retransmitted? \$\endgroup\$ – us2012 Jul 29 '13 at 16:15
  • \$\begingroup\$ Brian has the answer, although I had a follow-up question. To answer your questions, its to be a standalone system. I want 1-2 uSec latency tops, and I want the "matrix" to be able to handle terrabytes. Data is received and transmitted on 1Gb ethernet. \$\endgroup\$ – Brooks Jul 29 '13 at 16:47
  • 1
    \$\begingroup\$ Few elaborations required: what do you mean by "matrix of smart boards"? Are the boards independent? Is the memory shared between boards, or each board has its own memory? In general, if you're going to implement standalone application which does not require "plug-and-play" functionality, I see no reason to employ PCIe. There are simpler protocols which will have lower latency. As to the bandwidth, what do you mean by saying "terrabytes"? \$\endgroup\$ – Vasiliy Jul 29 '13 at 17:08
  • 1
    \$\begingroup\$ "Terabytes"? Terabytes of RAM? Terabytes per second? Terabytes of address space? Why 1-2 uS of latency, when you will have much more than that over Gigabit Ethernet? BTW: A terabyte of server RAM is about US$10K, but there isn't an off the shelf motherboard that can take it. \$\endgroup\$ – user3624 Jul 29 '13 at 17:14
  • 1
    \$\begingroup\$ Have you looked into Serial Rapid I/O? \$\endgroup\$ – vicatcu Jul 29 '13 at 17:39
2
\$\begingroup\$

This sounds a lot like how high-throughput routers are designed. One of the main problems in router design is exactly the "memory matrix" or "cross-bar backplane" that you describe. The large companies in the space (Juniper, Cisco, F5, etc) do not use PCIe for this purpose.

I think a better interconnect might be HyperTransport (which is AMDs solution to the same problem that Intel later developed as QuickConnect for.)

Note that in a PC, the PCIe bus sits "after" a HyperTransport bus, and thus HT has lower latency and higher throughput than PCIe can get to RAM (which in turn sits on the other end of the CPU these days.)

\$\endgroup\$
3
\$\begingroup\$
  1. Yes, it is the motherboard processor is the main controller. Specifically the CPU or the "North Bridge" is the master of the PCIe bus. This is the same as PCI. However a PCIe device can access memory, but it needs to first request the bus from the PCIe master.
  2. Some PCIe chips can do DMA where they handle transfers by themselves. This is common for network cards. However there is overhead, so it will provide more throughput with larger transfers.

EDIT: PCIe tops out at 5 Gb/s/lane for PCIe Rev 3. It tops out at 16 lanes. That's a total of 80 Gb/s for a single card. That's nowhere near 1 Tb/s, let alone > 1 TB/s. Your goal is going to be difficult. But then I don't think you would want to use PCIe for a > 1 TB/s Ethernet switch. That's > 800 10 gigabit Ethernet ports.

\$\endgroup\$
  • \$\begingroup\$ I had forgotten about DMA. I think that's the answer. I can have the smart card construct a block of data and DMA it into the matrix. Now to find a way to tell the other card that its data has arrived. And, how does PCIe have terrabyte transfers if the memory operates at 133Mhz? \$\endgroup\$ – Brooks Jul 29 '13 at 16:50
1
\$\begingroup\$
  1. The "North Bridge" (a.k.a. motherboard chipset) is ultimately in control of the PCIe bus, with a lot of interaction with the main CPU and the BIOS/OS. Once configured, the northbridge, and any other PCIe switches, can do most things without having to involve the CPU.
  2. PCIe is really designed for transferring lots of data at a time using DMA. It can do single-word transfers, but it slows down a lot when doing that. Also the read latency is huge (~2 uS), so if you can't accommodate that with large packet sizes and DMA then total system performance can really suffer.

If throughput is even a minor issue, then you really need to plan on using DMA to move all of your audio.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.