0
\$\begingroup\$

I'm interested in wiring up some Raspberry Pi Compute Modules together on a custom PCB board with DDR2 SODIMM connector interfaces.

http://www.raspberrypi.org/raspberry-pi-compute-module-new-product/

I want to use the compute modules together in a N-body-type simulation where each module acts as a planet or something and interacts with all other modules via MPI. Ideally each module should be able to asynchronously communicate with any other module in the board without interrupting the processing on the other modules.

What is a good hardware solution to high-bandwidth switching for PCB components connected in this fashion? Normally with standard Raspberry Pis I would use an ethernet switch to route packets between the nodes but I'm not so sure here. Would it be feasible to build switching logic + processors directly into the board (or have several of the compute modules serve as switching logic)?

Some research seems to indicate that Blue Gene node cards (it has a similar design to the tray I am proposing above) are connected via tx+rx 12-channel optical fibers, but I'm not sure if that is what I need, as that seems to be missing some kind of switching capability.

Thanks!

\$\endgroup\$
1
\$\begingroup\$

How many modules are we talking about here? 10? 10,000?

There are a couple of possibilities, depending on what sort of I/O capability the boards have and how many boards you want to connect. I'm not aware of any obvious off-the-shelf solutions, however.

One method is to provide each module with one or more Ethernet PHY chips, and then use some commodity ethernet switch chips to connect them together, presuming the compute modules can interface well with a PHY chip. This could work well if a non-mesh network is OK (I don't think el cheapo ethernet switches can handle loops). If you want to do something a bit more complicated, you could make a bunch of small boards with a handful of compute modules, PHY chips, and an integrated switch, then connect the boards together with a larger off-the-shelf Ethernet switch. You would probably be limited to using gigabit ethernet, though.

Another possibility is to use some FPGAs. Inside of an FPGA there is a decent number of block RAMs that can be configured as dual port RAMs or FIFOs. If you have a relatively small number of compute modules, you can just connect them all to a single large FPGA with enough pins, then put togther an FPGA configuration with all of the interface logic as well as a bunch of interconnecting block RAMs and/or FIFOs. If you have a large number of modules, then what you can do is use FPGAs to build a mesh network with a small number of compute modules connected to each FPGA. This way you have complete control over the routing of data between modules. I would recommend using either parallel interconnections between the FPGAs or possibly serialized LVDS for more bandwidth, depending on if the FPGAs have transceivers and how crazy you feel like getting with the board layout.

\$\endgroup\$
  • \$\begingroup\$ Hi, thanks for the quick answer. One question - what do you mean by loops? If every compute module has to go through the integrated switch chip to get to another compute module, there wouldn't be any loop topologies in the network, right? BTW I'm planning on about ~128modules per board. \$\endgroup\$ – ejang Jun 29 '14 at 17:29
  • \$\begingroup\$ With a mesh network, it is possible for a packet to come back to the same node along many different paths. I do not think commodity single-chip switches are capable of properly dealing with this configuration. How many nodes are you planning on in total, and how much bandwidth will each node require? \$\endgroup\$ – alex.forencich Jun 29 '14 at 23:19
  • \$\begingroup\$ 128 nodes per board, gigabit ethernet switch connecting 32 boards in a rack (total 4096 nodes). Ideally, any node can send a short 100 byte message to any of the other 4095 nodes, and each node sends messages to 100 other nodes at least 10 times per second. I don't see how a mesh network is needed - won't all traffic simply be routed through a powerful central switch? Each board would have its own FPGA or IC acting as a switch between the 128 local modules in that board, but for communication between modules on different boards, traffic would flow from chip 1->board A->switch->board B->chip 2 \$\endgroup\$ – ejang Jun 30 '14 at 5:05
  • \$\begingroup\$ Ah, I see what you're getting at. 100 bytes * 100 nodes * 8 bits * 10 Hz = 800 kbps per node, and 102 Mbps per board. Definitely doable. The main issue becomes an interfacing issue; the SoC used on the raspberry PI is a POS and has terrible peripheral support. The simplest setup fo you (presuming 1 Mbps per node is acceptable) would be to use the serial or SPI (no USB Ethernet BS) to interface directly to FPGAs on the board for local routing (e.g. a bunch of Spartan 6 devices) and then hang a gigabit PHY off of one of the FPGAs (take a look at the Digilent Atlys board). \$\endgroup\$ – alex.forencich Jun 30 '14 at 21:49
  • \$\begingroup\$ cool, thanks. makes a lot of sense. In this case though, I'm planning on interfacing from SoC to the FPGA through the GPIO pins from the RPi compute module (which does not have any of the connectivity modules). \$\endgroup\$ – ejang Jul 1 '14 at 20:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.