3
\$\begingroup\$

I'm working on an embedded system which will generate data at a pretty high rate, 100-300Mbit/s. This is coming from a bunch of SPI busses. I need to get this into a Linux PC. What's the best way to do that?

At the moment I have a small portion of this built with a STM32F4 microcontroller as proof of concept. It gets data at around 20Mbit/s. I'm realizing that (a) there is probably no way for the F4 to read data at 100Mbit+, and that is probably still going to be hard/impossible for faster parts like F7 or H7 since the APB clock is limited to 100MHz; and (b) there is no way to send data to a PC that fast from these microcontrollers either, especially while both reading and sending. Reading and sending simultaneously at 20Mbit is probably (just barely) possible.

The main options for sending data to the PC would seem to be USB or ethernet. High speed USB is theoretically fast enough at 480 Mbit, but ST microcontrollers mostly don't have it - although it looks like some parts have ULPI. 100Mbit ethernet is not fast enough; gigabit is again theoretically fast enough, but none of the ST microcontrollers have that (or SGMII). I don't know what the practical speed ceiling is for streaming data over HS-USB or 100MBit ethernet, but I'm guessing it's a lot less than the theoretical speed.

So, the question is, what is the best way to stream data at the aggregate speeds I need? (either with a fast IO friendly microcontroller, or something else).

A few options:

  • split the device into several independent devices (maybe 4-8 or so) and send data from each one either over HS-USB with a ULPI USB chip (USB3300 or similar) or 100Mbit ethernet to PC with multiple USB ports/multiple ethernet ports - the PC can reassemble these streams, or
  • same, but put something in front to aggregate the connections: USB hub or ethernet switch. could be on board to the device reduce number of physical wires, or a separate device
  • use a different microcontroller or SOC which can handle the total data rate in one device
  • use a FPGA (but which one? it seems like most low cost FPGAs like ICE40 don't have SGMII either)

In this case there is literally no processing that happens; the micro sends a few setup commands to the SPI buses and after that it's just passing data through (possibly assembling the data into blocks and adding headers per block on the output). There are also no special timing/latency requirements, as long as the data can stream continuously with nothing dropped. I want a setup that is as simple and robust as possible. I'm very familiar with ARM microcontrollers; less so with FPGAs. I'd like to avoid high-end/specialty parts (of course a Kintex FPGA could handle this, but...)

\$\endgroup\$
3
  • \$\begingroup\$ No one knows what is "best" for you but you. You have some vague requirements (what does "high-end" mean? what does "but..." mean?) so this is pretty much an opinion question, and is therefore off-topic. \$\endgroup\$ – Elliot Alderson Mar 16 at 20:10
  • 2
    \$\begingroup\$ Would a USB3 ASIC work? I've used the fx3 at over 3 Gbit/s. \$\endgroup\$ – user1850479 Mar 16 at 21:32
  • 1
    \$\begingroup\$ A lot of the STM32 parts have high speed USB, they just need an external PHY. The STM32f723 does however have an internal high speed PHY. \$\endgroup\$ – Colin Mar 17 at 11:35
8
\$\begingroup\$

OK, after the first couple of lines you've, correctly, ruled out microcontrollers. So, you've ruled out microcontrollers!

What you need is something that, as you've noticed, "speaks" a fast interconnect on one side, and multiple SPI buses on the other.

The only way I see that happening is an FPGA.

Attaching FPGAs to Gigabit ethernet is pretty standard, and it's very likely that whatever FPGA you pick, its manufacturer has a ethernet MAC block ready to use that you "just" have to connect to your SPI controllers that you implement inside the FPGA, and write a bit of state machine to actually stream the data (many designs even implement a small CPU like a picoRV32 in the FPGA itself to control the streaming/framing). All you then need to do is attach a PHY, and an ethernet jack that contains the magnetics. Many FPGAs come with Evalboards that carry exactly that. An ECP5 might be a cheap entry here.

You're under the wrong impression that the FPGA would bring an SGMII interface by itself - no, the whole points of FPGAs is that you can configure their logic such that you build such an interface.

If you want to go the USB route, there's also (less common) USB2 controller blocks for FPGAs, and I know of at least one free&open source USB3 controller in development (Luna by Kate Temkin). However, the usual route would be attaching your FPGA to a specialized USB controller IC thing; Cypress' FX2 and FX3 are the "classics" here.

\$\endgroup\$
2
  • \$\begingroup\$ I was looking at simple/small FPGAs like ICE40 and Max5/10 - maybe I'm missing the "how" of it but it doesn't seem there's any way to build SGMII interface without transceivers? (although maybe RGMII is possible) Lattice ethernet IP core seems to be ECP2M/3/5 only latticesemi.com/en/Products/DesignSoftwareAndIP/… \$\endgroup\$ – Alex I Mar 16 at 22:48
  • \$\begingroup\$ Ice40, Max5 might really really be to small, and Max10 is... smaller than comfortable. Yes, you're missing the point. SGMII interfaces are just logic attached to one of the many IO pins of an FPGA. You can basically do that on any FPGA, given it's fast enough and given the voltage standards of the IO ports allow that (and RGMII voltage levels is exactly what FPGA IO pins are designed for...). You don't have to use the vendor-supplied IP blocks, it can be comfortable. I already recommended ECP5, so... \$\endgroup\$ – Marcus Müller Mar 17 at 10:36
4
\$\begingroup\$

STM32H7 would likely be fast enough to handle it. I have used it for a similar purpose, with data coming in at 100+ MByte/s, which was then realtime compressed to about 30 MByte/s that was transmitted over USB High Speed.

Using USB HS on STM32's is not particularly difficult. You just need the external PHY that is connected via the ULPI pins, and software driver that supports high speed mode. I used Chibios, and it was easy to get going. At that point in time it did not support DMA for the memory to USB peripheral transfers, but it didn't appear to be a bottleneck in my application.

Getting the data into the processor can be more problematic, as the interfaces are not as flexible as an FPGA. If the normal SPI peripherals are fast enough for your purpose, you can just use them with DMA. Interleaving data from multiple SPI peripherals can be done with DMA, or you can transmit them in separate packets and process on PC. DCMI can also be (ab)used to read up to 14 synchronous signals at up to 100 MHz. QuadSPI peripheral can manage 200 MHz reading in DDR mode, but it appears to have a transfer size limit of 4 GByte.

If you haven't done FPGA development before, getting USB communication running can be quite a challenge, even if you use a premade IP core for it. Contrary to other claims, I'm confident that 300 Mbit/s is manageable on a 480 MHz STM32H7, and does not even need very much optimization.

\$\endgroup\$
5
  • \$\begingroup\$ agreed, a sufficiently clocked cortex-M7 would do for the ethernet case, where it's not even necessary to react to any bus transactions but you can just fire ethernet packets and forget. I personally would put the STM32H7 somewhere between microcontrollers and application processors, but arm and ST label them as MCU, so I guess you're right. \$\endgroup\$ – Marcus Müller Mar 17 at 11:51
  • 1
    \$\begingroup\$ @MarcusMüller Well, I'm not so sure about ethernet case. At least STM32H7 does not have gigabit ethernet. But USB-HS is faster than 100Mbit ethernet. And usually ethernet requires more buffer memory, because TCP may need retransmissions after several milliseconds, unlike USB where the delay is 125 us for USB-HS. \$\endgroup\$ – jpa Mar 17 at 13:46
  • 1
    \$\begingroup\$ good point, but streaming-to-USB is usually harder than streaming-to-Ethernet, because in USB, you need to give data when the host asks, in Ethernet, you can send as long as the medium's not busy. The former typically comes with mean latency requirements, which might mean USB handling comes in the way of handling your SPI data. (really depends on how much DMA you can do before you really need to handle data) \$\endgroup\$ – Marcus Müller Mar 17 at 13:48
  • 2
    \$\begingroup\$ The USB peripherals handle that low level communication completely autonomously, and often have several kilobytes worth of FIFO space. So you just put data to FIFO, and the USB peripheral automatically sends the packets to host when it requests them, or replies with NAK if the FIFO is empty. \$\endgroup\$ – jpa Mar 17 at 13:50
  • \$\begingroup\$ ah cool! Yeah, then it's rather easy, and USB vs Ethernet should be decided on other factors (do you wish to use long cabling/networking? WHat's worse: needing a driver under Windows or configuring your network card?) \$\endgroup\$ – Marcus Müller Mar 17 at 13:56
2
\$\begingroup\$

I should use an FPGA. I don't think you need a high-end FPGA for this project. An Artix or Spartan 7 from Xilinx can easily handle that data. Ethernet is probably the easiest and best way to transfer data and since data speed is 100Mbps+ the 1GBe using RGMII is the way to go. The site FPGA Cores has a core that you can connect to an RGMII phy directly and you will be up and running in minutes. The cores are free to download. I have used these cores (mostly RMII/100Mb) and they are very easy to work with.

You could have a look at this example using a Mimas Artix board with Gigabit Ethernet.

\$\endgroup\$
1
  • \$\begingroup\$ I agree with this approach. I worked on an embedded project a while back that had a "fixed" MAC function. It basically took a stream of data, and chopped that up into ethernet frames, and would send those. There was no flow control. The incoming data rate was lower than the ethernet speed. An external PHY was used to adapt to the medium (this was 100BASE-TX, but 1000BASE-T or any other variant would be OK too). USB is fine, I think, but there is a lot of hair on it. Ethernet is relatively simple, IMHO. One tricky part of OPs design will be merging the streams without screwing the pooch. \$\endgroup\$ – Troutdog Mar 18 at 13:39
2
\$\begingroup\$

That really depends on how many SPI busses you got and what throughput you got on each.

If the number of SPI busses you got fits the number of SPI peripherals available on a microcontroller which has USB2 and an appropriately smart DMA engine, then this is the way to go. Clock speed is misleading ; for example a 204 MHz Cortex-M4 LPC4330 will do bulk USB2 480 Mbps just fine with a few percent CPU load on the M4, but that's because it has several rather smart scatter-gather DMA engines and ridiculous memory throughput (something like several gigabytes/s) due to AHB bus matrix and several memory banks that operate in parallel. You can use DMA to transfer from SPI to RAM, then the USB peripheral uses DMA again. The CPU never touches the data, it just shuffles pointers.

But does it have enough SPI peripherals for you? That's the main factor. Obviously, at the throughput you need, you're not going to bit-bang it. It absolutely needs hardware.

Also note I'm not particularly recommending the LPC4330, it's a great chip but it's a bit old and last time I checked (5 years ago) the provided software libraries were garbage, I had to write the USB driver myself. This is quite often the case: you get a micro with great hardware that can easily do 48 Mb/s USB2 without breaking a sweat, and the example code is a piece of junk that does 1 Mb/s. So another important thing to add to the checklist if you're in a hurry is "the libraries include easy to reuse example code that actually does 480Mbps USB2 bulk without crashing."

If you have not-so-fast SPI busses, like 30 Mbps each, but say 10 of them, so no microcontroller will ever have enough SPI channels. The simplest solution would be to plug each into an off-the-shelf FT232H USB2-to-SPI bridge. These will do 30Mbps each. Pros: problem solved, no hardware to design, just a little bit of software on the PC side. Cons: not for high volume production obviously, no synchronization between streams.

If you like sexy and exotic, there's XMOS. I think it can do the job. It's also quite easy to use, but there are not that many boards available.

Besides that, there's the FPGA solution. It shouldn't be too difficult on the SPI side, you can have as many channels as you have pins for them. On the USB side, a FTDI USB fifo chip or a Cypress FX2 chip are much easier and faster to get running than a USB core. Try to get a board that has a USB port and one of these USB chips on it, then double check the schematics to make sure the USB chip is not just used to program the FPGA, but also has a fast communication path with the FPGA.

I would only consider having the FPGA do the USB job if the board came with solid example code and a USB core that doesn't need a $$$ license.

Personally I wouldn't bother with ethernet, unless... as above, you get a board with solid example code that will do 1Gbps. USB is much simpler, and it has higher total throughput if you use multiple USB2 ports.

\$\endgroup\$
1
\$\begingroup\$

You could look at the MAX10 series FPGA from Intel/Altera. They're non-volatile FPGAs. Most high end FPGAs need an external part to store the configuration. I'm very entry level in FPGA. The learning curve is rather steep. Terasic (link below) has some development boards you could try depending on your budget.

There is also Xilinx FPGAs. I'm not familiar with them though.

https://www.terasic.com.tw/cgi-bin/page/archive.pl?Language=English&CategoryNo=13&List=Simple#Category218

\$\endgroup\$
1
\$\begingroup\$

You might want to check out the i.MX RT series by NXP. They are H7 ARM parts with BGA versions that run at 600MHz+ with an embedded HS USB PHY. Not quite sure they can handle the full 200-300 mbits you specified due to USB/processing overhead however.

FPGA is a route you generally don't want to take unless you are extremely familiar with HDL and all interfaces involved (timing, protocols, software stacks for USB/Ethernet/whatever you are implementing). From personal experience, I'd say spinning up an FPGA board takes about 2% of my time and the software + verilog + debugging takes up the other 98%, even when I am somewhat familiar with the protocols involved.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.