Some PCIe devices can map their own device memory region fully to contiguous host physical memory address space through a feature called PCIe Resizable BAR (base address register), which makes it possible to overcome the usual memory region size exposed by BAR.
Through this, it is possible to directly address all the PCIe device memory by the host user/kernel thread like normal host DRAM space.
I was able to test this concept by mapping the physical contiguous memory region for the PCIe device memory onto user space virtual memory address space, using RX560 on Linux through mmap(). As a result, memset(,0xff,)-ing its 4GB VRAM took 4 times more time compared to dual-channel 4-slot 2Rx8 DDR4-2133 host memory with AMD 5900X processor.
Which one is faster, DMA addressed by the PCIe device side or MMIO addressed by the host processor for both read and write (usually the DMA region is not that big since it needs contiguous host-side DRAM space for the same size as DMA region as the device does not understand host paging, but just for comparison), if we assume that all the memory write by host-side MMIO will be done in non-temporal fashion (directly through the write buffer) and data (device memory region) is considerably huge (like N GB). For DMA, the time cost to read from and write to the data into the host memory should be included, of course.
For example, if we compare 1) 4GB memset() on the host DRAM and issue DMA request to the PCIe device to copy it to its device memory and 2) 4GB memset() directly on the device memory physical address region, which one will be faster (and vice versa)? Is the host dram access plus the block transfer to/from PCIe device (I don't know whether the host memory controller and PCIe protocol supports it or not) faster than the sequential word-size memory accessing directly to the PCIe device?