Creating hardware acceleration card with PCIe implemented in an FPGA

Question

PCIe offers a very high speed interface between a peripheral card and PC. Sending and receiving data from the FPGA side which contains the PCIe IP seems to make sense. However, I am confused how it would work on the computer side. On the PC side we have different operating systems and they usually do not allow direct access to I/O ports.

How would one write a program on the PC side (lets say windows) to communicate with the PCIe peripheral card? I think most likely we would use C or C++ but don't know anything more than that.

Answer 1

If you use PCIe on an x86 system, you will need DMA in the endpoint. Otherwise your transfers will be quite slow, as x86 platforms don't tend to have host-side DMA resources. Reads, especially will be painful as they're non-posted and take a long overhead for each transfer.

In Xilinx-land, the most popular PCIe bridge block is called 'xdma', which is documented in PG195. This block supports memory-mapped or streaming DMA, and has drivers available for Linux and Windows. While it's not terribly complicated, you do need some kernel expertise to integrate it with your application. It's a well-known block, and there's lots of chatter on the Xilinx forums about it.

There's a couple of other IP blocks in Xilinx (like cdma and vdma), or you can purchase a block as IP from companies like Northwest Logic.

As far as how PCIe devices 'work', they are enumerated at boot time using I/O, and configured by the driver to live in a specific set of memory address spaces, called 'resources' in Linux and BARs in PCIe. The way you communicate with a device is you ask for its virtual address when you 'open' it, then you can read/write to that address. That address isn't fixed: it changes as it's assigned by the MMU each time you open the device.

If you want to see how that works, the utility 'pcimem' (Linux) shows a very basic way to do that. It's a simple program that does peek/poke to any PCIe device.

Answer 2

You just found out you need a driver to communicate with hardware. Indeed!

And that driver will depend on what you do on the FPGA. Your FPGA vendor provides examples for examples of things you can do!

Answer 3

You would either need to create a device driver or use/modify an existing driver. It also depends on whether you use the "bare" PCIe IP core, or something slightly higher-level such as RIFFA, XDMA, or QDMA. If you use the bare PCIe IP core, you'll probably want to write your own driver. If you use something higher-level, it may come with its own driver, which you can then interface with from your own application software.