2
\$\begingroup\$

Looking at the PCIe DMA solution offered by different FPGA vendors, there are 2 main user-interface options:

1) AXI Memory Map (Altera use Avalon-MM) 2) AXI Streaming (Altera use Avalon-ST)

Using PCIe in FPGA world, may i know which user interface is most commonly used? My interpretation from Xilinx is AXI Streaming is meant for those subsytem developed by pure RTL based while AXI memory map is meant for full AXI sytem design (I assume it refer to SOC / ARM / Microblaze subsystem)

In other words, can i say that AXI memory map is not suitable to the RTL-based design at all?

\$\endgroup\$
1
\$\begingroup\$

I would say that it heavily depends on the application. It's certainly possible that a memory-mapped interface could make a lot of sense even without a CPU or SoC. It's also possible that a streaming interface could make sense even with a CPU or SoC. It's also possible both are needed.

Memory-mapped interfaces are necessary if the target is memory, be it on-chip SRAM, off-chip DRAM, or perhaps HBM. Streaming interfaces can be convenient for applications that involve streaming data, such as networking, image and video processing, DSP, etc. You certainly don't have to have a CPU to have a use for a large bank of DRAM.

However, the design of the DMA core, the interfaces it provides to FPGA logic, as well as the operation of the PCIe bus can possibly cause some problems if very high performance is required.

One major problem is PCIe ordering. PCIe supports multiple outstanding DMA read operations, but it makes no guarantees about the order in which those reads complete. This can cause serious issues with cores that provide streaming interfaces, as either the core can only make a single read request at a time (resulting in poor performance) or the core must spool the read data in onboard RAM so it can get things organized, which increases complexity and resource utilization and can affect performance.

Another problem is alignment. Data is transferred over the PCIe bus DWORD aligned to host memory. Inside the FPGA, the alignment will be different - either address aligned on a memory-mapped interface, or cycle aligned on a streaming interface. Matching the alignment on a TLP by TLP bases can require inserting a large number of wait states, which reduce throughput. The headers attached to each TLP also affect throughput - even though the TLP max payload size is an even power of two, adding some headers means the overall size is not an even power of two, and this can result in a lot of empty byte lanes between TLPs, decreasing throughput, even more so as the bus width increases.

I'm wrestling with some of these issues myself in https://github.com/ucsdsysnet/corundum . There is no processor core in this pure Verilog design, but the (fully custom) DMA core uses a memory-mapped AXI interface to efficiently deal with interleaved completions. I am currently in the process of moving from an AXI interface to a segmented memory interface to increase the throughput over the PCIe link as the current AXI-based datapath has some serious shortcomings.

\$\endgroup\$
7
  • \$\begingroup\$ For the PCIe ordering issue occur in streaming interfaces, is it something can be resolved by with using Xilinx multiple DMA channels configuration? such as having having multiple DMA engines to improve the performance. \$\endgroup\$
    – Learner
    Oct 3 '19 at 2:12
  • \$\begingroup\$ I understand that you are not using Xilinx DMA IP & build your own one from RTL. But i'm just hoping to get some hint on how to optimize the performance with leveraging the existing features/components the FPGA vendor offers. \$\endgroup\$
    – Learner
    Oct 3 '19 at 2:16
  • \$\begingroup\$ Do you think if anyone can build the PCIe & AXI memory map in RTL based? I mean compare to the streaming interface, probably user will choose to go for slightly easier way while still maintaining reasonable system performance. \$\endgroup\$
    – Learner
    Oct 3 '19 at 2:20
  • \$\begingroup\$ I have no idea what the XDMA core does internally. Possibly it has some block RAM to get things organized. Multiple interfaces may or may not help depending on how the core is built. Again, Xilinx really does not say much about what's in the core, so you might just have to benchmark it. Multiple DMA engines will probably cause more problems than they solve, though it is necessary to use one DMA engine per PCIe interface, so if you have a card with a bifurcated x16 edge connector you'll need to use two DMA engines to get full bandwidth. \$\endgroup\$ Oct 3 '19 at 2:56
  • \$\begingroup\$ Xilinx also has the newer QDMA core which is supposed to be pretty high performance, but I have not used it personally. The device support for that core might also be limiting. \$\endgroup\$ Oct 3 '19 at 2:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.