1
\$\begingroup\$

I am trying to understand the PCI-E principles but I miss something. The reason why I need to understand that is because I working on a project that involve a lot of SATA HDD to be connected to one machine. Because the HDDs are heavy and occupy a lot of space I decided to divide the hosting machine in multiple slaves.
So in my idea one "master" that is equipped with a middle range processor and a mid class motherboard will host 16 HBA PCI-E boards. The "master" server machine have only a SATA SSD for OS and intended to use 4x 1to4 PCI-E riser (splitter) attached to PCI-E slots from motherboard, one riser with his own PCI-E slot.
On the "slave" side we have a computer M-ATX case equipped with 32 HDD, powered by a 650W gold PSU and the riser board with 4 slots of PCI-E equipped with 4 HBA boards. Slave is connected to the master with USB cable provided in the riser package. In theory each PCI-E from the master have 4x HBA cards, and those are connected to 32 SATA HDD.
The idea works until I connect the 4'th slave to 4'th PCI-E. So there is no problem with 3 PCI-E connected with 1to4 PCI-E and host 12 HBA boards (96 HDD). I read about the "lanes" that PCI-E use it, about how chipset and processor use them, but I think I have understood in the wrong way the principle.
I have tried to connect this slave to multiple "master" configurations, and I have used as a master the following configurations:

  • HP Z800 (equipped with dual processor x5675 that intel report it have 32 PCI-E lanes), motherboard used on this system use intel 5200 chipset that reports it can carry 72 PCI-E lanes)
  • HP Z600 (tested before Z800, almost the same config as Z800)
  • Z570 motherboard with Ryzen 5900X (AMD reports 32 PCI-E lanes for processor and 36x usable for AMD chipset Z570)
  • Entry level chipset H110 (6 PCI-E) and B450 (4 PCI-E) with Intel gen 4 and 6 processor.

    All of those configurations fail to boot up to the BIOS when I connect the 4'th slave. Master is powered and on HP six beeps keep report me an error "HP says to check the video card", on the rest of configurations master is power on but there is no signal from video card. I have changed the order of slaves, boards, added another riser between them, powered the risers from the same PSU as the master, more and more, but I believe the problem is a limitation that I don't understood it.

If someone can help me to understand what is happening here and why my setup is not working in this situation, I beg it to do it, because my mind is blowing after 20H of tries. I have attached a picture made in paint, maybe will help to understand the setup.
Thanks !
Flow image

\$\endgroup\$
3
  • \$\begingroup\$ This site isn't really about your question. It's a Q and A site for EE design questions and not about proprietary electronic products where design information is lacking. I appreciate the diagram but it's still off-topic I believe. \$\endgroup\$
    – Andy aka
    Commented Sep 25, 2021 at 10:29
  • \$\begingroup\$ @Andyaka first time I have posted on SO, they told me to post here because the topic is about electronics principle. There is not any problem if i need to delete the post, but where you think I need to post ? Thanks \$\endgroup\$
    – sniper
    Commented Sep 25, 2021 at 14:12
  • \$\begingroup\$ I have no idea where to post. \$\endgroup\$
    – Andy aka
    Commented Sep 25, 2021 at 17:53

1 Answer 1

Reset to default
1
\$\begingroup\$

PCIe "lanes" aren't dedicated throughout the fabric, in the same way that traffic lanes aren't dedicated throughout a city. You can have for example an x4 (4 lane) link to a bridge with 16 x1 downstream ports and you can use all the ports simultaneously. Overall throughout is limited by bottlenecks in the same way traffic through a city is. Lane count is not a limitation in your design. Look instead to BIOS limitations or device/hardware limitations (the risers). Also keep a close eye on the power supply! Sounds underpowered to me. If they're traditional platter disks instead of SSDs, remember they have high inrush current to spin up and if not staggered they will require a much bigger supply just to handle the start.

\$\endgroup\$
5
  • \$\begingroup\$ PSU cannot be underpowered because is not used to power the risers, the slaves where I have the HDD and the riser board have their own PSU, the power consumption is about 330W (total for 3v,5v and 12V), each slave have seasonic gold rated psu 650W (stressed out to 600W at the start (for 6-8 seconds) and works great). master have same power supply only for the MB and processor and stays under 100W, risers are powered on from the slaves. You are right about the power if the riser take power trough the PCI-E port, but they have 4x molex input for power and I have populted them from the slaves psu \$\endgroup\$
    – sniper
    Commented Sep 25, 2021 at 14:10
  • \$\begingroup\$ Ok, I was back to the office to test your theory with the underpowered. Added to one PCI-E at master, another 1to4pci riser, powered on another PSU just for this riser, and populated 3 PCI from the riser with slaves (12 HBA), I have measured the power consumption with and without HBAs and the conclusion was that the psu from master don't consume any W for the pci-e I connected the HBAs, all the power for those are from secondary PSU. Same thing 3 slaves connected to the secondary riser works, when I connect 4'th slave (to riser slot or separate pcie from master) not work anymore. \$\endgroup\$
    – sniper
    Commented Sep 25, 2021 at 16:43
  • \$\begingroup\$ @sniper Fair enough - in that case I don't have an answer for you, but suggest you look into the hardware specs in detail, and maybe check with the manufacturers for support. But don't worry about lane counts. PCIe is a switched fabric: with only a single x1 link from the root complex you can bridge out to an essentially unlimited number of devices - at least as far as PCIe itself is concerned. \$\endgroup\$
    – TypeIA
    Commented Sep 25, 2021 at 16:57
  • \$\begingroup\$ @sniper Actually my first suspect would be the BIOS. It may have limits on endpoint count and topology. The BIOS provider would know best and if you're very lucky the limits if any might be documented somewhere. Note this wouldn't be any physical limit arising from PCIe, rather a limit imposed by the BIOS firmware, specifically with regard to enumeration. \$\endgroup\$
    – TypeIA
    Commented Sep 25, 2021 at 17:02
  • \$\begingroup\$ Yes, I think the BIOS do this limitation. That's why I have tried multiple configurations to test multiple BIOS vendors. All of them manifest the same problem. I will try to obtain more details about BIOS limitations to the PCI-E. Can't upvote your response because I am a newbie. "Thanks for the feedback! You need at least 15 reputation to cast a vote, but your feedback has been recorded." But thanks for your response ! \$\endgroup\$
    – sniper
    Commented Sep 25, 2021 at 17:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.