1
\$\begingroup\$

I have an ambiguity regarding the PCIe initial configuration which is performed by the root complex (RC) on the end-points (EP).

Both devices have their own base address registers (BARs). The RC configures an address on the EP BAR. What does that address represent?

Also, how does the EP know the value of the RC BARs to initiate some read or write transactions with the RC as only RC can send configuration read/write requests? I have read the specification document but still don't fully understand the mechanism.

Clarity on the above would be greatly appreciated.

\$\endgroup\$
4
  • \$\begingroup\$ A warm welcome to the site. This answer may help your understanding. \$\endgroup\$
    – TonyM
    Mar 22 at 11:02
  • \$\begingroup\$ @TonyM Thank You! I have gone through your answer it explains the configuration very well. My question was more focused towards bus mastering as I have gathered from the answer below which was how does the EP know the address of the RC if it needs to initiate transfer of some data to it (DMA etc.) \$\endgroup\$
    – malik12
    Mar 22 at 11:10
  • \$\begingroup\$ You're very welcome. Answering your comment question: The RC hardware and all BARs form a single memory map. Any PCIe agent being an initiator is programmed with addresses valid for that memory map, usually obtained from the OS. So the initiator doesn't need to understand the memory map, it's given the addresses to to use, derived from addresses provided ultimately by the governing software (OS). \$\endgroup\$
    – TonyM
    Mar 22 at 13:22
  • \$\begingroup\$ @TonyM Got it, thanks again! \$\endgroup\$
    – malik12
    Mar 22 at 13:56

2 Answers 2

2
\$\begingroup\$

When an endpoint has some memory - be that physical RAM, control/status registers, Flash, etc. - the system needs some way of knowing how to address that memory. Remember that software running in the system uses memory addresses to access data. By assigning a physical address to the device, the memory can be accessed by software just like any other memory.

The address programmed into the BAR on an endpoint represents the physical memory address in the system that software can use to communicate with the corresponding memory in the endpoint. An EP can have up to 6 of these regions. The lower N bits of the BAR will always be 0, where N is the width of the memory. For example if you had a 64kWord memory on your EP, that would be a 16b wide word address, so the lower 16b of the BAR will be hardwired to 0's.

The EP doesn't necessarily know the address values of the RC. It doesn't initially need to. EPs start as being passive devices - only responding to requests from the RC. If they need to send their own write and read requests, the OS must configure them to be capable of performing bus mastering.

Once capable of bus mastering, software on the system (typically device driver) will configure the device to initiate requests - for example if you wanted to DMA a chunk of data from say system RAM to your device or vice versa, the driver would work out the physical memory address it is required to communicate with and send that information to the EP as part of setting up an operation.

If you have two PCIe devices that are intended to talk directly to each other (P2P), again it would be up to the device driver or other such software to read the BAR addresses for the two EPs and pass that information to both devices so that they know how to address each other.

\$\endgroup\$
3
  • \$\begingroup\$ Thank you for your detailed response, in the last paragraph where you say "read the BAR addresses for the two EPs and pass that information to both devices so that they know how to address each other" can you please specify where exactly does an EP store the address of the other EP or RC it needs to communicate with? \$\endgroup\$
    – malik12
    Mar 22 at 10:13
  • \$\begingroup\$ @malik12 That would be up to the device itself. They could use one of the BAR regions for some extra custom configuration registers to store such info. \$\endgroup\$ Mar 22 at 12:57
  • \$\begingroup\$ Ok I think I get it now, thanks again. \$\endgroup\$
    – malik12
    Mar 22 at 13:53
2
\$\begingroup\$

There is only a single address space, and it is decoded from the root complex downwards.

The root complex acts as a bridge between the platform bus and the PCIe domain below it, so the addresses programmed into the RC are the (memory, IO and bus) ranges that exist in this domain, anything included in the range is expected to be connected to a downstream port, anything not included is connected elsewhere -- so this is basically the routing table for address-based routing.

Endpoints can initiate (bus/device/function routed) messages to other devices in the same domain, and (address routed) transactions to any destination in the system, including main memory and other devices.

Driver software is responsible for writing working addresses into hardware registers, for example you could have a register pointing to a DMA buffer, and that DMA buffer could be located on another device (in which case the address is inside the ranges programmed into all bridges above that device, so the access is routed correctly).

The mechanism for storing another device's (bus/device/function or memory) address is not specified in PCI(e), because that will depend on the requirements of the application.

\$\endgroup\$
3
  • \$\begingroup\$ Ok so there is only a single address space and multiple devices have their own physical memory which are the ranges programmed in the EP BAR. So the address for DMA communication from EP to RC are communicated on a separate method as opposed to programming the BAR? and if so then what is the purpose of the two RC BARs? is it for the host device only? \$\endgroup\$
    – malik12
    Mar 22 at 12:14
  • \$\begingroup\$ @malik12, Yes, the addresses used for EP-initiated transfers (which can go to the RC or any other EP in the same domain, i.e. connected to the same RC) are communicated separately. The RC BARs have the same function as the two BARs in PCI and PCIe bridges, they tell the bridge what addresses are connected below. The RC is like any other bridge in that regard. Any BARs, including those on the RC, are only written by the host. \$\endgroup\$ Mar 22 at 13:14
  • \$\begingroup\$ Ok, thank you for your detailed response. \$\endgroup\$
    – malik12
    Mar 22 at 13:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.