I am new to PCIe. I would like to understand 256 (bus), 32 (device), 8 (function). I am trying to visualise these PCIe slots on a motherboard. I am used to desktop motherboards where we have one PCIE_16, two PCIE_8, and two PCIE_2 slots.
How can a motherboard contain 256 buses (slots)? How can 256 buses and 32 devices be connected to a motherboard?
Those are the maximum address values. Your standard desktop motherboard will typically use very few of those addresses.
For example if all of your PCIe slots come off the same switch/root/bifrucated port in the CPU, then all of the slots will likely have the same bus - usually 0 - with each card you plug in having a device number starting from 1 (device 0 is the upstream port/CPU).
Most cards plugged in will likely have one function only, so have function aIfddress of 0. If you have a gfx card with audio (e.g. hdmi port), it will then likely have two or more function addresses, e.g. function 0 for graphics and function 1 for audio.
Your motherboard will probably then have another one or more other buses each with their own bus address for connecting other devices on the motherboard such as audio controller, storage controller, usb controllers, and so forth.
Where the large bus space comes into use is when you start having lots of PCIe switches with multiple levels. Each downstream port of the PCIe switch (or group of ports) will have its own bus number with the port of the switch being device 0 on that bus, then connected devices having device addresses of 1 or more.
With a multi layer tree of switches, you can very quickly use up the bus address space.
In addition to the other answer, if PCIe hotplug is used then bus space needs to be reserved to allow for PCIe devices to be enumerated when they are hot-plugged, i.e. connected after the BIOS and/or Operating System have performed an initial scan.
If there is no free bus space, then after hot-plugging a PCIe device, you can get errors such as devices behind bridge are unusable because [bus 04] cannot be assigned for them when performing a PCIe bus scan after hot-plugging a device:
[ 237.231357] pci 0000:03:00.0: [10b5:8111] type 01 class 0x060400
[ 237.231527] pci 0000:03:00.0: supports D1
[ 237.231529] pci 0000:03:00.0: PME# supported from D0 D1 D3hot
[ 237.231652] pci 0000:03:00.0: disabling ASPM on pre-1.1 PCIe device. You can enable it with 'pcie_aspm=force'
[ 237.231666] pci 0000:03:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 237.231699] pci_bus 0000:04: busn_res: [bus 04] end is updated to 04
[ 237.231703] pci 0000:03:00.0: devices behind bridge are unusable because [bus 04] cannot be assigned for them
[ 237.231707] pcieport 0000:02:00.0: bridge has subordinate 03 but max busn 04
[ 237.231911] pci 0000:07:00.0: PCI bridge to [bus 08]
The above errors were from a 4.18.0-372.26.1.el8_6.x86_64 Linux Kernel in Alma Linux 8.6, and are from 7.3. Unable to assign bus errors with AlmaLinux 8.6 when attempted a rescan after powering on the PEX8311 card, which are some of my notes about how to (re-)load PCIe devices after Linux has booted.
I haven't used any Thunderbolt (interface) hardware, but on a quick search think Thunderbolt can allow a hierarchy of PCIe switches. The Intel Thunderbolt Technology Brief contains:
Key Features:
- Daisy chain up to six devices
A Thunderbolt controller contains:
- A PCI Express switch with one or more PCI Express protocol adapter port
Haven't found an example of what the PCIe bus topology looks like when Thunderbolt devices are daisy-chained.
