PCIe End-Point configuration for communication with Root Complex

Question

I have an ambiguity regarding the PCIe initial configuration which is performed by the root complex (RC) on the end-points (EP).

Both devices have their own base address registers (BARs). The RC configures an address on the EP BAR. What does that address represent?

Also, how does the EP know the value of the RC BARs to initiate some read or write transactions with the RC as only RC can send configuration read/write requests? I have read the specification document but still don't fully understand the mechanism.

Clarity on the above would be greatly appreciated.

Answer 1

When an endpoint has some memory - be that physical RAM, control/status registers, Flash, etc. - the system needs some way of knowing how to address that memory. Remember that software running in the system uses memory addresses to access data. By assigning a physical address to the device, the memory can be accessed by software just like any other memory.

The address programmed into the BAR on an endpoint represents the physical memory address in the system that software can use to communicate with the corresponding memory in the endpoint. An EP can have up to 6 of these regions. The lower N bits of the BAR will always be 0, where N is the width of the memory. For example if you had a 64kWord memory on your EP, that would be a 16b wide word address, so the lower 16b of the BAR will be hardwired to 0's.

The EP doesn't necessarily know the address values of the RC. It doesn't initially need to. EPs start as being passive devices - only responding to requests from the RC. If they need to send their own write and read requests, the OS must configure them to be capable of performing bus mastering.

Once capable of bus mastering, software on the system (typically device driver) will configure the device to initiate requests - for example if you wanted to DMA a chunk of data from say system RAM to your device or vice versa, the driver would work out the physical memory address it is required to communicate with and send that information to the EP as part of setting up an operation.

If you have two PCIe devices that are intended to talk directly to each other (P2P), again it would be up to the device driver or other such software to read the BAR addresses for the two EPs and pass that information to both devices so that they know how to address each other.

Answer 2

There is only a single address space, and it is decoded from the root complex downwards.

The root complex acts as a bridge between the platform bus and the PCIe domain below it, so the addresses programmed into the RC are the (memory, IO and bus) ranges that exist in this domain, anything included in the range is expected to be connected to a downstream port, anything not included is connected elsewhere -- so this is basically the routing table for address-based routing.

Endpoints can initiate (bus/device/function routed) messages to other devices in the same domain, and (address routed) transactions to any destination in the system, including main memory and other devices.

Driver software is responsible for writing working addresses into hardware registers, for example you could have a register pointing to a DMA buffer, and that DMA buffer could be located on another device (in which case the address is inside the ranges programmed into all bridges above that device, so the access is routed correctly).

The mechanism for storing another device's (bus/device/function or memory) address is not specified in PCI(e), because that will depend on the requirements of the application.