5
\$\begingroup\$

There may be several points in this question which may not make sense to a knowledgeable person, so I do thank you for your understanding in advance.

For a computer vision project I wish to process images from a camera array (10s of cameras) simultaneously and we wish to jam image data to GPU memory as fast as possible, with RDMA or NVIDIA's gpuDirect. There exist cameras with SDI outputs and SDI capture pci-e cards with said direct GPU memory access, however, a setup like this may end up being quite expensive.

From the cost aspect USB cameras we are considering seem to be a solution, however, I am yet to find a USB host pci-e card that can support RDMA or gpuDirect.

(1) Have any of you seen a product that can support such direct access schemes? (Maybe something that emulates driver of the camera one end and can directly write to GPU memory via pci-e)

(2) Is what I am looking for even technologically possible? When I contacted a USB host pci-e card manufacturer I have received the following response that I cannot even interpret:

All our USB 3 controllers follow the xHCI specification published by Intel: https://www.intel.com/content/www/us/en/io/universal-serial-bus/extensible-host-controler-interface-usb-xhci.html

When data is transferred from a usb device, the controller is given a list of address/length pairs where the data should be written. The the controller will issues a PCIe MEM_WR to the address it is given with the data. I hope that helps you decide if you can do what you want with our host controller

So do you guys think it is possible or not (By possible I do not mean designing my own custom usb host capable of only doing what have described but with what's available already in market)?

\$\endgroup\$
3
  • 1
    \$\begingroup\$ docs.nvidia.com/cuda/gpudirect-rdma/index.html \$\endgroup\$
    – gustavovelascoh
    Oct 24, 2018 at 10:11
  • 2
    \$\begingroup\$ I do thank you, however, if you can narrow, what you want to share, down to a specific section it would be more helpful, as I have already visited this. \$\endgroup\$
    – keoxkeox
    Oct 24, 2018 at 10:31
  • \$\begingroup\$ The interpretation of manufacturer's response is that in standard xHCI architecture the controller's DMA (linked-list parsing) engine is given address areas in CPU memory space, and all USB data are placed (and dispatched from) there. If the GPU memory can be shared/embedded in the main CPU space (I have no idea if it is possible), it should be possible to re-design the USB xCHI driver to DMA its data into re-allocated area. Still the CPU must maintain all these crazy structures in memory for the xHCI controller to operate, CPU must have unrestricted access to the GPU memory. \$\endgroup\$
    – Ale..chenski
    Oct 25, 2018 at 6:16

1 Answer 1

Reset to default
5
\$\begingroup\$

Haven't heard of this, but looking in GPUdirect description and developer site, it seems like it is needed that the device driver for the 3rd party device, should be capable of assign GPU address to the device, in order to perform DMA operations (sections 1.2 and 1.3).

I am not used to PCIe, but I guess that as you plan to use USB cameras, they will be communicating with the USB driver in your system. So 2 options I see are:

  1. The one you stated, build your own USB host with RDMA capabilities, or
  2. modify/create a USB driver for your cameras including Nvidia Kernel driver features, to allow these features to be used from a CUDA program.
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.