PCI express devices are required to support MSI: what does that mean?

Question

I am a bit confused concerning an issue I had with a PCI Express (PCIe) card used to interface serial device on a Linux computer. The card is based on the XR17V354 chip from MaxLinear.

From the beginning, the card was correctly recognized by the OS (it was listed in the lspci output), but reading and writing to the serial device (/dev/ttySx or /dev/ttyXRx) failed, no matter which driver was used (8250 serial driver built into the Linux kernel or device/chip manufacturer driver).

Now, I eventually traced back the problem to an interrupt issue: interrupts were correctly raised on the card (visible by polling the UART ISR register and also in the PCI INTx register), but never "reached" the CPU (counter in /proc/interrupt stuck at 0).
The issue was solved by disabling MSI (Message Signaled Interrupts), by adding pci=nomsi in the Linux kernel command line parameters.

Now, this is the part that confuses me:

• I read that PCIe devices are required to support MSI

PCI Express permits devices to use these legacy interrupt messages, retaining software compatibility with PCI drivers, but they are required to also support MSI or MSI-X in the PCI layer.

• In the chip datasheet and also in the lspci -v output, I see that there is an entry in the PCI Configuration Registers for the MSI capability (MSI Capacity ID = 0x05).
• However, all the associated registers (eg Message Control Register) are Read-Only... which explains why MSI is not working and I had to disable it in the OS so it would handle INTx interrupts raised by the device

Does it mean the device is "broken", ie not compliant to PCIe requirements?
Or does the MSI support required from PCIe devices just implies to have a MSI Capacity ID (0x05), but doesn't actually require to implement the MSI feature?

Answer 1

The PCI Express specification says:

MSI/MSI-X interrupt support, which is optional for PCI 3.0 devices, is required for PCI Express devices. All PCI Express device Functions that are capable of generating interrupts must support MSI or MSI-X or both.

It also adds:

MSI and MSI-X are edge-triggered interrupt mechanisms; neither the PCI Local Bus Specification, Revision 3.0 nor this specification support level-triggered MSI/MSI-X interrupts. Certain PCI devices and their drivers rely on INTx-type level-triggered interrupt behavior (addressed by the PCI Express legacy INTx emulation mechanism). To take advantage of the MSI or MSI-X capability and edge-triggered interrupt semantics, these devices and their drivers may have to be redesigned.

Many manufacturers want to avoid the redesign effort, especially if they need to support legacy interrupts for older operating systems.

There are tests to check for PCIe compliance. However, a generic test cannot actually run the device (because most devices behave differently), so to check for MSI support, all it can do is to search for the presence of the MSI registers. This implies that the simplest way to pass such a PCIe compliance test is to add a bunch of non-working MSI registers. (The driver then must never try to enable MSI.)

So yes, your device violates the PCIe specification. (I guess that manufacturer driver simply added new PCI device IDs to the standard driver, and was never tested on a system that actually supports MSI.)

Answer 2

PCI Express components are one of the largest sources of non-compliance I have come across.

Even though MSI is mandated by the specification, many parts do not actually provide the capability for a few reasons.

1. Legacy code often relies on the INTx physical signalling (INTA, INTB, INTC, INTD) and this type of part is designed to meet that specific need. This doesn't really break the part as INTx signalling is always available in PCI express anyway (although it is a bit naughty).

2. MSI has some performance gains but the logic to enable it requires more than just the MSI considerations - MSI must not go through while a DMA is in progress to prevent a race condition for instance. The performance gains on a quad serial port simply were probably not worth the effort.

This particular part has been around since 2009 (see the revision table) and non-compliant parts for this and other reasons (1) were very common at the time.

In the revision table, you will also find this gem (probably inserted because they had complaints about MSI not working):

1. It was common for Gen 1 devices to use the speed advertisement field (used during initial link partner training) for uses other than speed advertisements which caused me quite a few headaches when I was starting to use Gen 2 switches as they did not properly report only gen 1 speeds; switch manufacturers (PLX at least when they were known by that name) provided work-arounds for this sort of thing. For Gen 1 devices there was only one speed and they did not foresee their devices still being used in the (far off to them) days of Gen 2 and up.