NVIDIA Jetson custom carrier board does not booting when PCIE device is attached. How can I track the problem?

Question

I have designed an custom NVIDIA Jetson carrier board based on P3449-B01 Jetson Nano Carrier Board reference design by NVIDIA.

The carrier board is working fine, the linux is booting up, however when I attach an x1 PCIE device (NVME on x1 PCIE lane) the linux is not booting up.

I tried to track down the issue with journalctl but when I use the following command: journalctl --list-boot

it does not list the failed boot sequence.

My testing procedure:

1. Power the device from external DC source
2. Wait until my router assigns IP to the device
3. login into linux through SSH over ethernet
4. check dmesg or do any kind of testing

If I do these steps without PCIE device, all steps can be done. If I do it with the NVME attached it stucks at step 2. Router does not assign IP.

I have tested the on board PSU, it is tested to drive 2.5-3A current from 5V, and it is capable to drive upto 5A. In idle without NVME the NVDIA draws approx. 400mA current, with NVME it draws 700-800mA so I don't think it is PSU issue. Based on the current consumption, it tries to do something when the NVME is connected, but I cannot connect to the device to check dmesg...

I have double checked the wiring of the PCIE lines, all looks OK to me. PCIE lanes are routed differentially, matched to 90ohms on a 4 layer board. My last thinking probably it is related to signal integrity issue, but in this case still I should have a working linux system.

Any idea how to track down the PCIE issue?

UPDATE:

For suggestion of Ron Beyer I examined the problem on the debug serial terminal, where the boot messages can be read. With the SSD when it starts the kernel it drop plenty of error messages for the PCIE bus. Such as:

[    1.558693] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0008(Receiver ID)
[    1.581631] pcieport 0000:00:01.0:   device [10de:0fae] error status/mask=00000001/00002000
[    1.600096] pcieport 0000:00:01.0:    [ 0] Receiver Error         (First)
[    1.629938] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0008(Receiver ID)
[    1.645772] pcieport 0000:00:01.0:   device [10de:0fae] error status/mask=00000001/00002000
[    1.658393] pcieport 0000:00:01.0:    [ 0] Receiver Error         (First)
[    1.682121] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0008(Receiver ID)
[    1.695462] pcieport 0000:00:01.0:   device [10de:0fae] error status/mask=00000081/00002000
[    1.705945] pcieport 0000:00:01.0:    [ 0] Receiver Error         (First)
[    1.706543] pcieport 0000:00:01.0:    [ 7] Bad DLLP
[    1.708031] nvme 0000:01:00.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=0100(Transmitter ID)
[    1.708799] nvme 0000:01:00.0:   device [126f:2263] error status/mask=00001000/0000e000
[    1.709177] nvme 0000:01:00.0:    [12] Replay Timer Timeout
[    1.718745] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0008(Receiver ID)
[    1.721020] pcieport 0000:00:01.0:   device [10de:0fae] error status/mask=00000001/00002000
[    1.721495] pcieport 0000:00:01.0:    [ 0] Receiver Error         (First)

And after couple of secs these repeated errors are replaced with:

[    2.133600] pcieport 0000:00:01.0:   device [10de:0fae] error status/mask=00000001/00002000
[    2.136428] pcieport 0000:00:01.0:    [ 0] Receiver Error         (First)
[    2.138200] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0008(Receiver ID)
** 4 printk messages dropped ** [    2.144891] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0008(Transmitter ID)
** 15 printk messages dropped ** [    2.158376] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0008(Receiver ID)
** 9 printk messages dropped ** [    2.165170] pcieport 0000:00:01.0:    [ 0] Receiver Error         (First)
** 7 printk messages dropped ** [    2.171578] pcieport 0000:00:01.0:   device [10de:0fae] error status/mask=00000001/00002000
** 8 printk messages dropped ** [    2.182061] pcieport 0000:00:01.0:    [ 0] Receiver Error         (First)
** 8 printk messages dropped ** [    2.196782] pcieport 0000:00:01.0:    [ 0] Receiver Error         (First)
** 8 printk messages dropped ** [    2.205280] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0008(Receiver ID)
** 6 printk messages dropped ** [    2.221748] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0008(Receiver ID)
** 7 printk messages dropped ** [    2.229780] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0008(Receiver ID)
** 9 printk messages dropped ** [    2.248932] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0008(Receiver ID)
** 6 printk messages dropped ** [    2.256010] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0008(Receiver ID)
** 5 printk messages dropped ** [    2.263474] pcieport 0000:00:01.0:    [ 0] Receiver Error         (First)
** 3 printk messages dropped ** [    2.269535] pcieport 0000:00:01.0:    [ 0] Receiver Error         (First)
** 3 printk messages dropped ** [    2.278270] pcieport 0000:00:01.0:    [ 0] Receiver Error         (First)
** 6 printk messages dropped ** [    2.284579] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0008(Receiver ID)
** 11 printk messages dropped ** [    2.296427] pcieport 0000:00:01.0:   device [10de:0fae] error status/mask=00000001/00002000
** 11 printk messages dropped ** [    2.312921] pcieport 0000:00:01.0:   device [10de:0fae] error status/mask=00000001/00002000
** 4 printk messages dropped ** [    2.313943] pcieport 0000:00:01.0:    [ 0] Receiver Error         (First)
** 9 printk messages dropped ** [    2.333645] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0008(Receiver ID)
** 9 printk messages dropped ** [    2.352748] pcieport 0000:00:01.0:    [ 0] Receiver Error         (First)

I guess that is the PCIE bus error which comes from the PCIE lanes wiring?

Answer 1

A new prototype is working after the redesign of the PCIE lanes! x4 PCIE lane implemented for NVME connection. On a 4 layer PCB design, with through hole vias.

The modifications added:

• the length of the individual tracks of a differential pair is matched with 0.5% error
• the length of two different differential pairs is not matched. E.g. there is a 15% difference in length of PCIE0 RX diff pair and PCIE0 TX diff pair
• non-pcie signals are not crossing the PCIE lanes. They go around the the PCIE tracks
• length match is implemented on both sides of the individual tracks, in one differential pair, close to the host connector. And NOT segment-by-segment basis (in EAGLE there is no efficient way for segment-by-segment basis).
• vias were needed to implement routings for some traces however the number of vias are minimized. One track could have only two vias. One placed right after the pad of the connector, the other is placed right after the pad of the other connector, or the pad of decoupling cap of the TX line.
• vias in differential pair were placed in a symmetrical fashion
• vias hole size are less than 14mil (0.3683mm)
• at each via on the PCIE tracks has a GND via pair close to the via
• constant spacing were maintained along the full length where it is feasible
• TOP signals were referred to second layer GND, the BOTTOM signals were referred to third layer GND. On TOP and BOTTOM layers the GND was not filled close to the PCIE traces.
• Approx. x10 of signal spacing distance were kept between PCIE differential pairs where it was feasible. x4 distance was the smallest distance where the x10 cannot be maintained.
• RX signals were run in one group (close to each other), TX signals were run in other group (far from RX group)
• standard stack-up was used by the PCB manufacturer (Multi-CB). And the manufacturer has reference values for track width and spacing to reach 90 ohm

yay