1
\$\begingroup\$

PCIe spec defines 3 address spaces:

  • Memory
  • IO
  • Configuration

I can configure the BAR register to specify the memory address range that a PCIe device will claim.

How does a PCIe device know that its configuration space can be accessed through the memory address as well, i.e. MMCFG?

Or in other words, how to enable MMCFG for a PCIe device and tell the device where its configuration space is mapped to?

\$\endgroup\$
2
\$\begingroup\$

The memory mapping is an implementation detail inside the root complex, the card is sent CfgRd and CfgWr TLPs. The destination address information inside the TLP is filled out from the address used in the ECAM access, and the completion reply is translated back into a memory access result when it is received, by matching the tag field in the Cpl TLP.

The configuration space accesses are routed by the bridges, which are aware of the device numbers assigned to devices below them, and will forward the TLP only to the correct port.

The first TLP that arrives in a card will be a configuration access TLP, which contains the card's device number as the destination address, and the card will assume that the request was routed correctly and use that address going forward.

\$\endgroup\$
3
\$\begingroup\$

Taking the Base Address Registers as an example, these are either 32bit or 64bit (combined BAR0/BAR1, etc.) registers which as the name suggests are initialised with an address which points to the start (base) of a memory, set by root complex (RC) during initialisation.

How do you set the length though? There is no length register. Instead your device hardwires enough LSBs in the register to zero. The first thing the RC does is write to all BARs in all devices a value of 0xFFFF-FFFF-FFFF-FFFF. It then reads back the BAR values, which in the case of say a 256 byte BAR would read back 0xFFFF-FFFF-FFFF-FF00 because the lower 8 bits are masked by your device to always be zero. For a 64kB bar it would read back with 0000 at the end. For a 4GB bar it would read back with 00000000 at the end.

The RC will then assign an address to the BAR which is guaranteed to be aligned to at least the size of the BAR (a 256 byte BAR will never be assigned an address that isn't aligned to at least 256 bytes - usually it will be aligned to 4kB or the page size if smaller than it).

This address is a physical address within the system. Whenever another device (e.g. CPU) need to access the memory your BAR describes, it uses this address to do so. When a TLP is sent to your device, the lower bits will be the address within the memory, and the upper bits will always equal the BAR value. To work out which BAR was being addressed, you simply mask enough LSBs (8 for a 256byte memory) and compare the resulting value against the BAR value to see if it matches.


For configuration information, all PCIe devices support a "PCI Compatible" configuration space of 256 bytes which are always located as the first bytes within the device - the RC can simply send a read request to address 0 ofthe device to access this configuration space. No special setup required, all devices must have it. This space includes details such as supported link speed, lane widths, error information, link training statuses, and device capabilities.

PCIe devices require more space than this compatible header to provide additional configuration and capability information. To acheive this, the configuration header includes an 8-bit address field called the "Capabilities Pointer" which stores an additional set of PCIe specific capabilities, again within the first 256 bytes of the devices address space.

Device manufacturers then want even more space to add their own custom configuration and capabilities information to inform their drivers how to operate and what features are available. This is acheived using a linked list approach. The PCIe capabilities block has a field called the "Next capabilities pointer" which gives the memory offset within the device for the first manufacturer specific capabilities structure. Each capabilities structure has a pointer to the next allowing them to be placed anywhere within the first 4kB address space within the device.

The configuration space is typically accessed using configuration read and write TLPs. If the CPU supports ECAM (Enhanced Config Access Mechanism), the configuration information can be accessed by software without requiring an understanding of the configuration space through the CPU treating the configuration space as a flat memory map across all devices. Each device is given an address within this space whereby the address used by software has the bus, device, and function address information embedded within it.

\$\endgroup\$
2
  • \$\begingroup\$ Thanks. I can understand that the memory address used for MMCFG access must encode the B/D/F/offset information. But how is such address routed to the proper device? Or claimed by the proper device? I guess there must be some decoders along the way that will decode/route such requests so the proper device can receive then passively. Or there must be some mechanism that a PCIe device can claim such requests actively. But I am not sure which approach is used in reality. \$\endgroup\$ Nov 4 '20 at 9:44
  • 1
    \$\begingroup\$ @smwikipedia the B/D/F are programmed by the RC, the device doesn't have to encode /request/claim anything. During enumeration the RC basically assigns addresses to everything. Every bus in PCIe is point to point, so all it has to do is say to the one device connected to it, "you're address 1". If that's an endpoint, done. If its a switch it then tells the switch to check its first downstream port for connectivitiy, and assigns an address to it. This chain keeps going until every device has an address \$\endgroup\$ Nov 4 '20 at 10:12
0
\$\begingroup\$

I just found this in the PCIe spec $7.2.2 Enhanced Configuration Access Mechanism (ECAM):

The size and base address for the range of memory addresses mapped to the Configuration Space 15 are determined by the design of the host bridge and the firmware. They are reported by the firmware to the operating system in an implementation-specific manner. The size of the range is determined by the number of bits that the host bridge maps to the Bus Number field in the configuration address.

So it seems only the host bridge, which represents the host processor to initiate configuration transactions, needs to know the base address of MMCFG range. And I think it will use this info to identify an address targeting the MMCFG range and decode the B/D/F/Offset info from that address. And then generate and route the configuration transaction towards the proper PCIe device/function.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.