2
\$\begingroup\$

Some PCIe devices can map their own device memory region fully to contiguous host physical memory address space through a feature called PCIe Resizable BAR (base address register), which makes it possible to overcome the usual memory region size exposed by BAR.

Through this, it is possible to directly address all the PCIe device memory by the host user/kernel thread like normal host DRAM space.

I was able to test this concept by mapping the physical contiguous memory region for the PCIe device memory onto user space virtual memory address space, using RX560 on Linux through mmap(). As a result, memset(,0xff,)-ing its 4GB VRAM took 4 times more time compared to dual-channel 4-slot 2Rx8 DDR4-2133 host memory with AMD 5900X processor.

Which one is faster, DMA addressed by the PCIe device side or MMIO addressed by the host processor for both read and write (usually the DMA region is not that big since it needs contiguous host-side DRAM space for the same size as DMA region as the device does not understand host paging, but just for comparison), if we assume that all the memory write by host-side MMIO will be done in non-temporal fashion (directly through the write buffer) and data (device memory region) is considerably huge (like N GB). For DMA, the time cost to read from and write to the data into the host memory should be included, of course.

For example, if we compare 1) 4GB memset() on the host DRAM and issue DMA request to the PCIe device to copy it to its device memory and 2) 4GB memset() directly on the device memory physical address region, which one will be faster (and vice versa)? Is the host dram access plus the block transfer to/from PCIe device (I don't know whether the host memory controller and PCIe protocol supports it or not) faster than the sequential word-size memory accessing directly to the PCIe device?

\$\endgroup\$
1
  • \$\begingroup\$ PCIe is a packet-bases protocol, so ultimately both types of writes do exactly the same thing: send packets to the device with the data to be written. I suspect which approach is faster is going to depend heavily on how efficient software/hardware is at generating those PCIe packets using each method. \$\endgroup\$ Nov 22, 2022 at 2:24

1 Answer 1

1
\$\begingroup\$

The main question for performance is the allowed transaction size for the access you're going to make.

Generally, there are two classes of memory space: prefetchable and non-prefetchable. For the former, any access size is allowed, for the latter, the addressed device defines the permissible types.

If you declare your MMIO space to be non-prefetchable, the root complex will generate exactly the size of access that the CPU instructed it to do -- if you access 64 bits, there will be one packet with exactly 8 bytes payload length.

For prefetchable MMIO space, accesses may be coalesced, which is a significant speedup, but requires the receiver to be able to be more flexible about transfer sizes and ordering.

In the other direction, DMA packets have the same problem: DMA to non-prefetchable addresses needs to take into account what access sizes are allowed. Fortunately, all of host RAM is prefetchable, and we can issue read requests of up to 4 kB (but not crossing a page boundary) and write requests up to the maximum packet size.

My expectation would be that DMA from/to memory would pretty much always win for bulk speed simply because we can maximize the transfer size and thus minimize overhead and idle times.

There is another pitfall here: on Intel/AMD, cache coherency between the PCIe root complex and the CPU is guaranteed, so issuing a DMA read request will flush caches first (or fetch the data directly from the caches). This is a potential source of slowdown or speedup, depending on which of these happen, and other architectures do not have the same guarantee and require an explicit flush on the CPU side -- so when you write a driver, make sure to use the appropriate synchronization primitive even if it is a no-op on Intel/AMD.

The requirement to have contiguous memory for DMA is not real: PCIe does not allow a transfer to cross a 4kB page boundary anyway. The only thing you gain from contiguous memory is simpler bookkeeping inside the device. You can reduce the effort a little bit by getting larger buffers with greater alignment, e.g. by requesting 2 MB or 1 GB hugepages, but the larger you go the less likely it is that the OS can actually fulfill your request, and you need a fallback mechanism then.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.