43
\$\begingroup\$

I've got into a discussion in the comments of https://security.stackexchange.com/questions/109199/is-physical-security-less-important-now-for-securing-a-server?noredirect=1#comment194327_109199

The question is simple. Has anyone experience of successfully hotplugging a PCIe card? Does it require special motherboards and cards, or is it supposed to work on all consumer hardware?

\$\endgroup\$
2
  • 2
    \$\begingroup\$ The answer should be two-fold. Both hardware and software (its drivers) should support hot-plugging. \$\endgroup\$
    – jippie
    Dec 31, 2015 at 14:26
  • 1
    \$\begingroup\$ I dont know if this helps, but I just successfully removed the second passed-through GPU from a kvm windows machine without affecting the first gpu (the screen just flickered for a second). \$\endgroup\$
    – feedc0de
    Jan 20, 2018 at 19:57

5 Answers 5

69
\$\begingroup\$

I used to design PCI-Express hardware that required full hot-plug support in hardware and software, and it certainly is possible, but it's quite involved and requires extensive software support -- the hardware is actually quite simple. I had to design the hardware, then implement BIOS (UEFI) and kernel (Linux) support for hot-plugging arbitrary PCIe devices over fiber and copper.

From a software point of view, one must remember that PCIe continues with the PCI software model, including the concepts of bus, device, function addressing. When the PCI bus is enumerated, it's done as a breadth-first search: PCI Bus topology from tldp.org

PCIe enumeration is generally done twice. First, your BIOS (UEFI or otherwise) will do it, to figure out who's present and how much memory they need. This data can then be passed on to the host OS who can take it as-is, but Linux and Windows often perform their own enumeration procedure as well. On Linux, this is done through the core PCI subsystem, which searches the bus, applies any quirks if necessary based on the ID of the device, and then loads a driver who has a matching ID in its probe function. A PCI device is ID'd through a combination of it's Vendor ID (16-bits, e.g. Intel is 0x8086) and Device ID (another 16-bits) -- the most common internet source is the PCI ID Repository.

The custom software part comes in during this enumeration process and that is you must reserve ahead of time PCI Bus numbers, and memory segments for potential future devices -- this is sometimes called 'bus padding'. This avoids the need to re-enumerate the bus in the future which can often not be done without disruption to the system. A PCI device has BARs (base address registers) which request to the host how much and what type (memory or I/O space) memory the device needs -- this is why you don't need jumpers like ISA anymore :) Likewise, the Linux kernel implements PCIe hotplug through the pciehp driver. Windows does different things based on the version -- older versions (I think XP) ignore anything the BIOS says and does it's own probing. Newer versions I believe are more respectful of the ACPI DSDT provided by the host firmware (BIOS/EFI) and will incorporate that information.

This may seem pretty involved and it is! But remember that any laptop / device with an ExpressCard slot (that implements PCIe as you can have USB-only ExpressCards) must do this, though generally the padding is pretty simple -- just one bus. My old hardware used to be a PCIe switch that had another 8 devices behind it, so padding got somewhat more complicated.

From a hardware point of view, it's a lot easier. GND pins of the card make contact first, and we'd place a hot-swap controller IC from LTC or similar on the card to sequence power once the connection is made. At this point, the on-board ASIC or FPGA begins it's power-up sequence, and starts to attempt link-training its PCI Express link. Assuming the host supports hot-plugging and the PCI Express SLTCAP/SLTCTRL register (in spec: PCI Express Slot Capability Register, PCI Express Slot Control Register. There is a 1 and 2 for this as well -- enough bits to split across two regs). for that port was configured to indicate the port is hot-plug capable, the software can begin to enumerate the new device. The slot status (SLTSTA, PCI Express Slot Status Register) register contains bits that the target device can set indicating power faults, mechanical release latch, and of course presence detect + presence changed.

The aforementioned registers are located in 'PCI (Express) Configuration Space', which is a small region of the memory map (4K for PCIe) allocated to each potential bdf (bus:device:function). The actual registers generally reside on the peripheral device.

On the host side, we can use PRSNT1#/PRSNT2# as simple DC signals that feed the enable of a power switch IC, or run to GPIO on the chipset / PCH to cause an IRQ and trigger a SW 'hey, something got inserted, go find it and configure it!' routine.

This is a lot of information that doesn't directly answer your question (see below for the quick summary), but hopefully it gives you a better background in understanding the process. If you have any questions about specific parts of the process, let me know in a comment here or shoot me an email and I can discuss further + update this answer with that info.

To summarize -- the peripheral device must have been designed with hot-plug support in mind from a hardware POV. A properly designed host / slot is hot-plug capable as well, and on a high-end motherboard I would expect it to be safe. However, the software support for this is another question entirely and you are unfortunately beholden to the BIOS your OEM has supplied you.

In practice, you use this technology anytime you remove/insert a PCIe ExpressCard from a computer. Additionally, high-performance blade systems (telecom or otherwise) utilize this technology regularly as well.

Final comment -- save the PDF that was linked of the Base Spec, PCI-SIG usually charges bucks for that :)

\$\endgroup\$
6
  • 7
    \$\begingroup\$ And to top-off the security discussion, with a relatively cheap FPGA (like a Cyclone IV GX) acting as a PCIe device, your host machine is done -- the FPGA can perform whatever DMA actions it wants. \$\endgroup\$ Dec 31, 2015 at 17:59
  • 1
    \$\begingroup\$ Great explanation. What happens when a Hot-Plug capable PCIe card gets swapped? On one hand, the OS must enumerate the PCIe topology again, seeing that a new device was inserted (it can't predict size of BARs/ amount of Buses that might be requested by the newly inserted device), but on the other hand - re-enumerating the system might not be possible without affecting the resources that were already assigned to existing devices in the topology... \$\endgroup\$ Jun 12, 2016 at 10:28
  • 2
    \$\begingroup\$ Yep, it gets tricky. So using ExpressCard (EC) as an example, one way I did it was to 'pad' the number of busses to support adding a device that might branch to even more devices; most BIOSes with a simple EC slot just pad it by one bus number (we used that slot to expand to many PCIe devices). Likewise, you can 'pad' the memory range possible for assignment there to support a variety of devices with a contiguous address range, same with IRQs. The OS (with/without ACPI) can then do what it will. It's actually "simple", but the complexities of SW layers in a modern machine makes it harder. \$\endgroup\$ Jun 13, 2016 at 16:52
  • 1
    \$\begingroup\$ Isn't PCIe enumeration actually a depth-first search? The base and limit registers are set up such that all devices below a given port must be enumerated before moving to the next port. \$\endgroup\$ Jan 19, 2019 at 3:23
  • \$\begingroup\$ I have a related question. Imagine a PCIe endpoint based on a FPGA plugged in a Windows system. Originally the FPGA is loaded from PROM, the system starts up, enumeration works and everything is fine. Then we reload the FPGA with a new bitstream, however the new bitsrteam contains exactly the same PCIe endpoint. Will it be enough to save the configuration space of the original PCIe and copy it into the new one to make it work without having to reboot the PC or having to implement full support for hot plugging? Thanks. \$\endgroup\$
    – mbmsv
    Feb 9, 2021 at 22:32
14
\$\begingroup\$

Provided the power state monitoring connections have been exposed to the connector by the upstream switch, and the pluggable unit has exposed these pins and is configured to use them properly and (as Jippie notes) the software can detect the hotplug event and respond properly, the answer is yes.

Generally, this capability is primarily used in server farms and data centres for hotplugging PCIe disks among other things; I am not sure that consumer equipment will be fully hotplug capable (it is, I understand, optional in the specification).

Keep in mind that providing the necessary hardware to support hotplug costs money (although the majority is within the PCIe endpoint, it still has to be set up, usually via an eeprom), it will not usually be offered in a price sensitive market.

Note that dynamically updating the PCI address map adds significant complexity to the PCI(e) driver; if a new device is inserted, then it has to be mapped into whatever bus it lives on, with the associated new address translations, but if a device is removed and then replaed with something different, it makes keeping track of PCI space addresses quite complex.

Without this complexity, the PCI subsystem is scanned once (at system reset) and remains static; no further effort required.

Here is the PCIe v3.0 Base Spec; see section 6.7 (page 514) on Hot Plug support. An example of a PCIe card which does support hot-plug can be seen here, courtesy of iocrest. It can be clearly seen that the shorter connector trace is routed: 2-port SATA III (6G) PCI-e Controller Card, Marvell 88SE9120 Chipset

However on this Axxon card, the shorter trace can clearly be seen routed to the adjacent one. On a physical level alone, this card cannot support hot-plug: MAP/950 1 RS232 Serial Port I/O Card for PCI Express (PCIe)

\$\endgroup\$
1
3
\$\begingroup\$

It is supposed to work on all PCIe compliant hardware, whether all consumer hardware is truly compliant is a good question, as I am not deep into PCIe spec to know about testing requirements and even then, do all retailers check the validity of the claim? I think hardly any do.

Much like the whole safety standards thing. Half (<-hyperbole?) the EE labels we have you can claim compatibility with, without having to have everything you make tested. Since hotplug stuff isn't life threatening I can't imagine people being more strict about it.

I, for one, have never tried it and seeing as my Clevo Laptop drove the desktop entirely out of my house, I'm not about to try it, since the GPU module in my laptop claims no hotplug capability and is too expensive without being Dave Jones and getting $$$ for the vid of an exploding GPU.

\$\endgroup\$
1
\$\begingroup\$

Yes, it works. I was able to get it working to hotplug a router chassis linecard (containing 10+ PCIe devices). The chassis have 16 hotplug-able cards. Any of the card can be plugged in or out randomly at run time without affecting the traffic operations on the other cards.

The complexity to make it work depends on the CPU environment. On an embedded CPU, the work is simply setting up static resources map and handling the connection change events by attaching and detaching PCI devices. On x86, it is much more involved because of the complexity in error handling and BIOS/OS interactions.

\$\endgroup\$
1
\$\begingroup\$

I haven't performed an actual physical hot-plug by removing / re-inserting a PCIe card, but have made use of hot-plug operations by re-loading a Kintex-7 FPGA over JTAG during development while the PCIe card remains fitted and the PC is booted. Where the Kintex-7 PCIe v2.0 x4 interface is connected to the PCIe slot.

This is using:

  1. A HP Z4 G4 desktop PC, with the Kintex-7 FPGA card fitted to PCIe slot 3 on the motherboard.
  2. Running Alma Linux 8.9 as the operating system, with Kernel 4.18.0-513.9.1.el8_9.x86_64.
  3. Using the vfio-pci Kernel driver to allow user space to write drivers for custom PCIe endpoints in the FPGA, without needing to write a custom Kernel module.
  4. The IOMMU is enabled, since are using vfio-pci.
  5. Secure boot is enabled. This doesn't affect the hot-plug functionality, but was done as part of checking vfio-pci can be used with secure boot enabled.

The initial problem was when the FPGA was reloaded while the PC was booted, the default HP BIOS settings treated the PCIe link going down at the point when the FPGA was re-loaded as a fatal error, which caused the BIOS to reboot the PC recording an error of the following form PCIe Surprise Link Down error detected on Slot 0. B:14 D:0 F:0 in the BIOS event log.

On looking at the BIOS options in the GUI, there were none for enabling hot-plug. However, the HP BIOS Configuration Utility (BCU) command line utility does allow the entire BIOS settings to be saved to a text file, and individual BIOS settings to be modified. On inspecting the list of BIOS settings found some related to hot-plug.

For slot 3 enabled Hot Plug and also set the number of Buses to 8 (which reserves space for PCI bus numbers as devices are hot-plugged):

C:\SWSetup\SP143621>BiosConfigUtility.exe /setvalue:"Slot 3 Hot Plug","Enable"
<BIOSCONFIG Version="" Computername="COMPUTERNAME" Date="2023/01/01" Time="19:26:02" UTC="0">
        <SETTING changeStatus="pass" name="Slot 3 Hot Plug" returnCode="0">
                <OLDVALUE><![CDATA[Disable]]></OLDVALUE>
                <VALUE><![CDATA[Enable]]></VALUE>
        </SETTING>
        <SUCCESS msg="No errors occurred" />
        <Information msg="BCU return value" real="0" translated="0" />
</BIOSCONFIG>

C:\SWSetup\SP143621>BiosConfigUtility.exe /setvalue:"Slot 3 Hot Plug Buses","8"
<BIOSCONFIG Version="" Computername="COMPUTERNAME" Date="2023/01/01" Time="19:26:33" UTC="0">
        <SETTING changeStatus="pass" name="Slot 3 Hot Plug Buses" returnCode="0">
                <OLDVALUE><![CDATA[0]]></OLDVALUE>
                <VALUE><![CDATA[8]]></VALUE>
        </SETTING>
        <SUCCESS msg="No errors occurred" />
        <Information msg="BCU return value" real="0" translated="0" />
</BIOSCONFIG>

After enabling Hot Plug for slot 3 in the BIOS settings, on re-loading the FPGA while the PC is booted:

  1. No longer results in the BIOS re-booting the PC due to a fatal error.
  2. Allows Linux to handle the FPGA being re-loaded with a different image with a new identity and different BARs.

E.g. the sequence:

  1. Power on the PC with the Kintex-7 FPGA booting from configuration flash. Bind the vfio-pci driver and the FPGA can be used over the PCIe bus.
  2. The FPGA design enumerated at power on has a single BAR:
    $ dump_info/dump_info_pciutils 
    Access method : linux-sysfs
    domain=0000 bus=15 dev=00 func=00
      vendor_id=10ee (Xilinx Corporation) device_id=7024 (Device 7024) subvendor_id=0002 subdevice_id=0003 header_type=NORMAL
      control: I/O- Mem+ BusMaster-
      physical slot: 3-1
      IOMMU group: 41
      Driver: vfio-pci
      bar[0] base_addr=b6800000 size=10000 is_IO=0 is_prefetchable=0 is_64=1
    
  3. Load a different FPGA design over JTAG.
  4. dmesg -wH on the PC outputs the following when the FPGA is re-loaded. This shows the Linux Kernel Hot Plug logic recognising the PCIe link goes down, followed by enumerating the changed FPFA design:
    [Dec30 15:48] pcieport 0000:14:00.0: pciehp: Slot(3-1): Link Down
    [  +0.000008] pcieport 0000:14:00.0: pciehp: Slot(3-1): Card not present
    [  +0.000041] vfio-pci 0000:15:00.0: can't change power state from D3hot to D0 (config space inaccessible)
    [  +0.000157] pci 0000:15:00.0: Removing from iommu group 41
    [  +0.033635] pcieport 0000:14:00.0: pciehp: Slot(3-1): Card present
    [  +0.136945] pci 0000:15:00.0: [10ee:7024] type 00 class 0x058000
    [  +0.000051] pci 0000:15:00.0: reg 0x10: [mem 0x00000000-0x000fffff 64bit pref]
    [  +0.000032] pci 0000:15:00.0: reg 0x18: [mem 0x00000000-0x0001ffff 64bit pref]
    [  +0.000031] pci 0000:15:00.0: reg 0x20: [mem 0x00000000-0x0000ffff 64bit pref]
    [  +0.000030] pci 0000:15:00.0: Max Payload Size set to 256 (was 128, max 256)
    [  +0.000007] pci 0000:15:00.0: enabling Extended Tags
    [  +0.000245] pci 0000:15:00.0: Adding to iommu group 25
    [  +0.000062] pcieport 0000:14:00.0: bridge window [io  0x1000-0x0fff] to [bus 15-1d] add_size 1000
    [  +0.000007] pcieport 0000:14:00.0: BAR 13: no space for [io  size 0x1000]
    [  +0.000001] pcieport 0000:14:00.0: BAR 13: failed to assign [io  size 0x1000]
    [  +0.000002] pcieport 0000:14:00.0: BAR 13: no space for [io  size 0x1000]
    [  +0.000001] pcieport 0000:14:00.0: BAR 13: failed to assign [io  size 0x1000]
    [  +0.000003] pci 0000:15:00.0: BAR 0: assigned [mem 0x387f80000000-0x387f800fffff 64bit pref]
    [  +0.000013] pci 0000:15:00.0: BAR 2: assigned [mem 0x387f80100000-0x387f8011ffff 64bit pref]
    [  +0.000012] pci 0000:15:00.0: BAR 4: assigned [mem 0x387f80120000-0x387f8012ffff 64bit pref]
    [  +0.000013] pcieport 0000:14:00.0: PCI bridge to [bus 15-1d]
    [  +0.000006] pcieport 0000:14:00.0:   bridge window [mem 0xb6800000-0xb6ffffff]
    [  +0.000005] pcieport 0000:14:00.0:   bridge window [mem 0x387f80000000-0x387fffffffff 64bit pref]
    
  5. Since are using the vfio-pci driver which doesn't automatically load, re-run the script to bind the driver. Where the script binds vfio-pci against any PCI device with a Xilinx vendor ID:
    $ ~/fpga_sio/software_tests/eclipse_project/bind_xilinx_devices_to_vfio.sh 
    IOMMU devices present: dmar0  dmar1  dmar2  dmar3
    [sudo] password for mr_halfword: 
    Bound vfio-pci driver to 0000:15:00.0 10ee:7024 [0002:0001]
    Waiting for /dev/vfio/25 to be created
    Giving user permission to IOMMU group 25 for 0000:15:00.0 10ee:7024 [0002:0001]
    0000:30:00.0 10ee:7011 [0000:0000] is already bound to driver vfio-pci
    
  6. The re-loaded FGPA design, with a different identity and BARs, is showing as enumerated and is usable:
    $ dump_info/dump_info_pciutils 
    Access method : linux-sysfs
    domain=0000 bus=15 dev=00 func=00
      vendor_id=10ee (Xilinx Corporation) device_id=7024 (Device 7024) subvendor_id=0002 subdevice_id=0001 header_type=NORMAL
      control: I/O- Mem- BusMaster-
      physical slot: 3-1
      IOMMU group: 25
      Driver: vfio-pci
      bar[0] base_addr=387f80000000 size=100000 is_IO=0 is_prefetchable=1 is_64=1
      bar[2] base_addr=387f80100000 size=20000 is_IO=0 is_prefetchable=1 is_64=1
      bar[4] base_addr=387f80120000 size=10000 is_IO=0 is_prefetchable=1 is_64=1
    

The PC used for the above test can dual boot into either Alma Linux or Windows 11. Haven't yet investigated hot-plug functionality under Windows as would need to look for the Windows equivalent of the Linux VFIO for creating user-space drivers for custom PCIe endpoints in FPGAs.

The above shows that Hot Plug needs to enabled on the slot for Linux to support hot-plugging of devices. Where support for Hot Plug on a slot can be motherboard and BIOS specific. This is reported as HotPlug+ in the SltCap against the PCIe root port to which the card to be hot-plugged is connected to. E.g. from lspci:

      SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise-
             Slot #3 PowerLimit 0.000W; Interlock- NoCompl-
\$\endgroup\$

Not the answer you're looking for? Browse other questions tagged or ask your own question.