0
\$\begingroup\$

I have designed an custom NVIDIA Jetson carrier board based on P3449-B01 Jetson Nano Carrier Board reference design by NVIDIA.

The carrier board is working fine, the linux is booting up, however when I attach an x1 PCIE device (NVME on x1 PCIE lane) the linux is not booting up.

I tried to track down the issue with journalctl but when I use the following command: journalctl --list-boot

it does not list the failed boot sequence.

My testing procedure:

  1. Power the device from external DC source
  2. Wait until my router assigns IP to the device
  3. login into linux through SSH over ethernet
  4. check dmesg or do any kind of testing

If I do these steps without PCIE device, all steps can be done. If I do it with the NVME attached it stucks at step 2. Router does not assign IP.

I have tested the on board PSU, it is tested to drive 2.5-3A current from 5V, and it is capable to drive upto 5A. In idle without NVME the NVDIA draws approx. 400mA current, with NVME it draws 700-800mA so I don't think it is PSU issue. Based on the current consumption, it tries to do something when the NVME is connected, but I cannot connect to the device to check dmesg...

I have double checked the wiring of the PCIE lines, all looks OK to me. PCIE lanes are routed differentially, matched to 90ohms on a 4 layer board. My last thinking probably it is related to signal integrity issue, but in this case still I should have a working linux system.

Any idea how to track down the PCIE issue?

pcie layout

UPDATE:

For suggestion of Ron Beyer I examined the problem on the debug serial terminal, where the boot messages can be read. With the SSD when it starts the kernel it drop plenty of error messages for the PCIE bus. Such as:

[    1.558693] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0008(Receiver ID)
[    1.581631] pcieport 0000:00:01.0:   device [10de:0fae] error status/mask=00000001/00002000
[    1.600096] pcieport 0000:00:01.0:    [ 0] Receiver Error         (First)
[    1.629938] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0008(Receiver ID)
[    1.645772] pcieport 0000:00:01.0:   device [10de:0fae] error status/mask=00000001/00002000
[    1.658393] pcieport 0000:00:01.0:    [ 0] Receiver Error         (First)
[    1.682121] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0008(Receiver ID)
[    1.695462] pcieport 0000:00:01.0:   device [10de:0fae] error status/mask=00000081/00002000
[    1.705945] pcieport 0000:00:01.0:    [ 0] Receiver Error         (First)
[    1.706543] pcieport 0000:00:01.0:    [ 7] Bad DLLP
[    1.708031] nvme 0000:01:00.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=0100(Transmitter ID)
[    1.708799] nvme 0000:01:00.0:   device [126f:2263] error status/mask=00001000/0000e000
[    1.709177] nvme 0000:01:00.0:    [12] Replay Timer Timeout
[    1.718745] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0008(Receiver ID)
[    1.721020] pcieport 0000:00:01.0:   device [10de:0fae] error status/mask=00000001/00002000
[    1.721495] pcieport 0000:00:01.0:    [ 0] Receiver Error         (First)

And after couple of secs these repeated errors are replaced with:

[    2.133600] pcieport 0000:00:01.0:   device [10de:0fae] error status/mask=00000001/00002000
[    2.136428] pcieport 0000:00:01.0:    [ 0] Receiver Error         (First)
[    2.138200] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0008(Receiver ID)
** 4 printk messages dropped ** [    2.144891] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0008(Transmitter ID)
** 15 printk messages dropped ** [    2.158376] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0008(Receiver ID)
** 9 printk messages dropped ** [    2.165170] pcieport 0000:00:01.0:    [ 0] Receiver Error         (First)
** 7 printk messages dropped ** [    2.171578] pcieport 0000:00:01.0:   device [10de:0fae] error status/mask=00000001/00002000
** 8 printk messages dropped ** [    2.182061] pcieport 0000:00:01.0:    [ 0] Receiver Error         (First)
** 8 printk messages dropped ** [    2.196782] pcieport 0000:00:01.0:    [ 0] Receiver Error         (First)
** 8 printk messages dropped ** [    2.205280] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0008(Receiver ID)
** 6 printk messages dropped ** [    2.221748] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0008(Receiver ID)
** 7 printk messages dropped ** [    2.229780] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0008(Receiver ID)
** 9 printk messages dropped ** [    2.248932] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0008(Receiver ID)
** 6 printk messages dropped ** [    2.256010] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0008(Receiver ID)
** 5 printk messages dropped ** [    2.263474] pcieport 0000:00:01.0:    [ 0] Receiver Error         (First)
** 3 printk messages dropped ** [    2.269535] pcieport 0000:00:01.0:    [ 0] Receiver Error         (First)
** 3 printk messages dropped ** [    2.278270] pcieport 0000:00:01.0:    [ 0] Receiver Error         (First)
** 6 printk messages dropped ** [    2.284579] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0008(Receiver ID)
** 11 printk messages dropped ** [    2.296427] pcieport 0000:00:01.0:   device [10de:0fae] error status/mask=00000001/00002000
** 11 printk messages dropped ** [    2.312921] pcieport 0000:00:01.0:   device [10de:0fae] error status/mask=00000001/00002000
** 4 printk messages dropped ** [    2.313943] pcieport 0000:00:01.0:    [ 0] Receiver Error         (First)
** 9 printk messages dropped ** [    2.333645] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0008(Receiver ID)
** 9 printk messages dropped ** [    2.352748] pcieport 0000:00:01.0:    [ 0] Receiver Error         (First)

I guess that is the PCIE bus error which comes from the PCIE lanes wiring?

\$\endgroup\$
7
  • \$\begingroup\$ Are the RX/TX pair matched impedance? How about length? Is there a ground plane between the top and bottom layers? It looks like you routed the RX pair under a lot of other pins/traces, you may be getting some coupling there. It also looks like you routed them parallel to some other traces on the NVME connector. \$\endgroup\$
    – Ron Beyer
    Jan 30, 2022 at 15:34
  • \$\begingroup\$ Yeah, there is one solid ground plane between top and bottom which does not have any traces on it. The traces matched to 90 ohms. And yeah I figured out lately the PCIE clock and RX signals are going parallelly. Probably there will be a redesign, but it would be good if I could boot into linux and see some the error messages that the PCIE causes. \$\endgroup\$
    – D_Dog
    Jan 30, 2022 at 16:26
  • \$\begingroup\$ Does the Linux installation exist on the NVME drive? If so you probably won't be able to get past the first stage bootloader if it isn't working... \$\endgroup\$
    – Ron Beyer
    Jan 30, 2022 at 17:30
  • \$\begingroup\$ No, linux is booting from the integrated emmc. The NVME is just a drive. As I stated, without NVME the carrier board + jetson card is booting up properly. \$\endgroup\$
    – D_Dog
    Jan 30, 2022 at 18:07
  • 1
    \$\begingroup\$ Do you have a serial console you can hook into to watch the boot process? \$\endgroup\$
    – Ron Beyer
    Jan 30, 2022 at 22:09

1 Answer 1

1
\$\begingroup\$

A new prototype is working after the redesign of the PCIE lanes! x4 PCIE lane implemented for NVME connection. On a 4 layer PCB design, with through hole vias.

The modifications added:

  • the length of the individual tracks of a differential pair is matched with 0.5% error
  • the length of two different differential pairs is not matched. E.g. there is a 15% difference in length of PCIE0 RX diff pair and PCIE0 TX diff pair
  • non-pcie signals are not crossing the PCIE lanes. They go around the the PCIE tracks
  • length match is implemented on both sides of the individual tracks, in one differential pair, close to the host connector. And NOT segment-by-segment basis (in EAGLE there is no efficient way for segment-by-segment basis).
  • vias were needed to implement routings for some traces however the number of vias are minimized. One track could have only two vias. One placed right after the pad of the connector, the other is placed right after the pad of the other connector, or the pad of decoupling cap of the TX line.
  • vias in differential pair were placed in a symmetrical fashion
  • vias hole size are less than 14mil (0.3683mm)
  • at each via on the PCIE tracks has a GND via pair close to the via
  • constant spacing were maintained along the full length where it is feasible
  • TOP signals were referred to second layer GND, the BOTTOM signals were referred to third layer GND. On TOP and BOTTOM layers the GND was not filled close to the PCIE traces.
  • Approx. x10 of signal spacing distance were kept between PCIE differential pairs where it was feasible. x4 distance was the smallest distance where the x10 cannot be maintained.
  • RX signals were run in one group (close to each other), TX signals were run in other group (far from RX group)
  • standard stack-up was used by the PCB manufacturer (Multi-CB). And the manufacturer has reference values for track width and spacing to reach 90 ohm

new layout

lsblk

lspci

files

yay

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.