How are BAR registers handled between end points in PCI Express?

Question

I have a question related to the PCI Express protocol. I managed to understand most of the features of the PCI Express protocol but could not entirely understand the enumeration process.

I know the root complex sets the BAR registers of every end point on the link, but how do end points learn BAR registers of other end points? Through the root complex somehow?

Here are a couple of assumptions I have made.

1. In a computer, I assume that the root complex is some sort of Host Bridge and the CPU and GPU are end points. Is that correct?
2. The enumeration process is done in the root complex and not in the end points. if that's the case, how do the end points know the other end points' bus, device, and function IDs?
3. Lets say somehow the end points know the bus, device, and function IDs. Do the end points use these IDs to read from the configuration registers to get the BAR addresses?

Answer 1

PCI Express is an evolution of PCI, itself an evolution of ISA bus. These have something in common: they are all based on Intel x86 architecture. They’re all also heavily influenced by Microsoft, who for their part nurtured and promoted ‘jumper-free’ configuration.

One feature of x86 architecture is that has a split I/O vs. memory address space. PCIe, like PCI before it, uses I/O space for card enumeration. Each card geographically addressed in I/O space. That is, each ‘slot’ or endpoint is decoded by the bridging logic into its own unique I/O map. Thus, the endpoints don’t ‘know’ about each other, at least at this level: they have non-overlapping I/O space maps from the host point of view.

Within that card I/O map lives the Configuration Space, which is standardized by the PCIe spec. It occupies between 256 and 4K bytes of space. The BAR registers are mapped into that space in known locations.

(Side note: sometimes you’ll hear Configuration Space called Configuration Information Space, or CIS. CIS was introduced with PCMCIA; PCI followed soon after with Configuration Space. They’re very similar in functionality and structure. ISA Plug ‘n’ Play is closely related to these, though it didn’t work nearly as well owing to the need to coexist with legacy non-PNP hardware.)

During enumeration the host scans all its possible geographical ‘slot’ I/O spaces to find cards. As it finds each one it dumps its Configuration information, parses it, and makes this information available for the kernel. This is done by the BIOS or its equivalent.

As startup continues, the kernel uses the BIOS-provided data to install drivers and set up each card’s registers for operation, mapping their BARs and their extents into memory space segments that don’t overlap, and setting up other slot resources like interrupts and power management.

The kernel uses the same BAR information to create entries in the TLB to map virtual addresses into physical addresses for the endpoints (more below.)

The takeaway is, cards don’t access each other’s Configuration Space, this dealt with by the BIOS and kernel. They can access each other’s memory space if the platform supports it (more on this below.)

The root complex is what you think it is: a specific kind of bridge. In the past it was a separate chip, as part of the ‘south bridge’. Nowadays it’s inbuilt into the CPU and closely linked to its on-chip fabric.

Where is the endpoint Configuration Space stored then? It lives in the endpoint’s PCIe block, and is usually initialized from a nonvolatile memory at power on. The loaded-up endpoint map that it presents upstream follows specific rules defined by the PCIe spec for the I/O register map, including the BARs. The host determines this map as it parses Configuration Space before it begins setting things up.

Finally, to use a card’s memory resource, you make a system call to tell the kernel you need it mapped, which tells the driver to create the TLB entry to map the physical BAR-defined space into virtual space. An example of how to do this can be found at https://github.com/billfarrow/pcimem.

If your card uses DMA (usually the case in PC platforms), anything it communicates with also needs to be mapped. Host memory for example needs to have DMA-reachable segments mapped in the root complex for the cards to use.

More about PCIe BARs here: https://stackoverflow.com/questions/30190050/what-is-the-base-address-register-bar-in-pcie

PCIe also has the ability to do peer-to-peer communication. This requires additional hardware in either the root complex or a switch. It’s possible (and common) to build large multiprocessor systems this way, even regarding PCIe as a kind of IP backbone.

Answer 2

Devices can send each other messages (which are sent to a specific 16 bit bus-device-function address), or memory accesses (sent to a physical address).

You'd instruct a device to address memory on another device the same way you instruct it to use DMA memory: with a device-specific mechanism. For a simple device, that might be a single register, for a more complex device, a data structure in memory that lists available DMA areas.

If a DMA area points to RAM, the device will use RAM, if it points to another device, it will talk to the other device. Completions use bus-device-function addressing.

At least in theory. In practice, a lot of the root bridges do not support device-to-device transfers, and attempting to send a memory access TLP from one device to the other generates an Unsupported Request in the root complex.

If an IOMMU is used, there might also be address translation involved, and you need to specifically query the PCIe subsystem driver for the bus address of a BAR from the point of view of another device, and if this lookup fails, device-to-device transfers are not supported.

What often does work are direct transfers between devices below the same non-root PCIe bridge.

The other questions:

1. The root complex contains one or more bridges between an unspecified platform bus and a PCIe bus. The CPU and memory are on the side of the platform bus, so these are not PCI(e) devices. The root bridge translates requests between PCI(e) and the platform bus. If there are multiple PCI domains in a system, accesses between domains are also translated, and bus-device-function addresses never cross that boundary.

2. Devices only need to know another device's bus-device-function address when generating a reply, such as a completion. The original request contains the information where to address the reply. For requests between domains, or where one end is not on PCI, the respective address will be the root complex (00:00.0).

3. No. Any communication is, at least initially, mediated through the host CPU.

Answer 3

Communication between PCIe devices is necessarily mediated by software. It is absolutely possible for PCIe devices to talk to each other, both via memory/IO reads and writes against BARs as well as via PCIe messages. However, since devices cannot issue configuration reads and writes (only the CPU/root complex can do this), it's not possible for a PCIe device to probe the configuration space of any other PCIe device in the system and access BAR registers, capabilities, etc. Host software must mediate this sort of configuration and provide the information in a device-specific way - either by writing it to the BAR space, writing it to config space, or writing it out to a buffer in host memory that the card can be configured to read from.

The most common way for devices to talk to each other is via peer-to-peer DMA, and this generally requires OS involvement as well in many cases due to the presence of an IOMMU. With an IOMMU, devices by default cannot access each other's BAR space if they're connected to different root ports. So, the driver will have to talk to the OS to set up the IOMMU so that one device can perform reads and writes against the BAR space of the other device. I suppose another thing that's not quite peer-to-peer DMA is to have a shared buffer in host memory that both devices can access. This similarly also requires involvement from the OS to allocate the buffer and configure the IOMMU such that both devices can access the buffer.

Addressing your assumptions:

1. The "root complex" is not a single component, it is a logical construct that contains the CPU, memory, chipset/PCH, and perhaps a few other pieces. A dual socket machine will also have a single root complex, consisting of both CPUs, their associated memory, chipsets/PCH, etc. The root complex features one or more PCIe root ports (which may live on the host CPU, chipset, etc.) through which PCIe devices can access system memory and the CPU can talk to the devices.

2. Devices do not know each other's IDs, unless they are informed by software running on the host, or if they receive a PCIe TLP from some other device.

3. No, the only component that can issue configuration operations is the root complex, PCIe endpoints cannot issue configuration requests. So, host software is required to read the BARs and provide this information to PCIe devices, if necessary.