I am thinking of creating a PCI-Express card to support from 1TB up to 8Terabytes of RAM (for servers). I will have to modify glibc in order to use it, but it is not a problem. The problem is to make it at lowest possible cost.

What integrated circuits can I use for cheap and reliable ECC DDR4 128GB memory modules? I am thinking to put 8 modules per PCI card.


closed as off-topic by PeterJ, Scott Seidman, Nick Alexeev Feb 14 '16 at 20:21

This question appears to be off-topic. The users who voted to close gave this specific reason:

If this question can be reworded to fit the rules in the help center, please edit the question.

  • 6
    \$\begingroup\$ the sentences "terabyte memory controller" and "FPGAs are expensive" in the next breath tells me you don't have any idea about the complexity of the problem you are trying to tackle. The cost of fpga develeopment tools would be neglible compared go even the cost of PCIE conformance testing and the fpga itself compared to TiB of ram modules per assembly. This is not something solvable with a $200 mcu dev kit and $200 in memory chips. This is a $100,000 project just to get a proto out \$\endgroup\$ – crasic Feb 13 '16 at 5:34
  • \$\begingroup\$ See superuser.com/questions/545378/… \$\endgroup\$ – David Feb 13 '16 at 8:25
  • \$\begingroup\$ Why don't you install as much memory as can fit in your server (should be at least 256GB), then buy 10 very fast SSDs and use them for swap space? And have you actually profiled your application to verify that the working set is really multi-terabytes? \$\endgroup\$ – Oleksandr R. Feb 13 '16 at 16:26
  • 1
    \$\begingroup\$ @nulik if it can be solved by a 2nd grade student why haven't you solved it? And no your guesses are wrong about what I do and where I work , the cost of compliance testing and development for pcie isn't just the final test you sent it off for to get the cert. The product doesn't cost 100k, just the development of an initial prototype will get close to that. More if you hire actual engineers. If you think the only gating item for this project is glibc you are deluded \$\endgroup\$ – crasic Feb 14 '16 at 2:45
  • 1
    \$\begingroup\$ It sounds like you fancy yourself an entrepreneur so let me deflate you a little. An entrepreneur who ignores and insults engineers, who has no clue about the complexity of the problem he wants to solve , doesn't understand the state of the art and his customer, who doesn't even recognize when he is out of depth is not an entrepreneur or innovator but a complete poser. This is a bad idea for a product. PCIE is not designed for this function and it is a kludge to even try. It's doomed to fail and doesn't pass the sniff test, but you think your shit don't smell \$\endgroup\$ – crasic Feb 14 '16 at 3:14

I think you may be biting off more than you can chew here.

First, what is the purpose of this card? Additional memory...for what? It can be interfaced over PCIe, true, but you'll never see the type of latency performance you could get from real DDR4 attached directly to the CPU memory controller. And why does it need to be DDR4?

Perhaps latency is less of an issue, and you can tolerate increased latency in exchange for having 1TB of volatile, RAM storage. You would have to use a FPGA to act as a PCI Express device that is connected to all this RAM. You are straight into Virtex-7/UltraScale or Stratix 10 territory, the flagship FPGAs from Xilinx and Altera respectively, each of which costs thousands of dollars a piece. They're the only parts with enough high-performance I/O to support what you need.

If you can find one of these FPGAs that can support the full amount of memory you need to interface too, great! Otherwise, you'll likely need to use a PCIe packet switch from the likes of PLX or let you strap several FPGAs behind a single x16 PCIe 3.0 link to the host machine. On these FPGAs, you'd have to instantiate the memory controller IPs, and develop / write an interface to go from RAM to PCIe.

So, this device is technically possible (the best kind of possible), but would be insanely expensive, power hungry, honestly not that fast, and potentially have issues fitting onto a full-length PCI Express card. Remember that you'd need to design a power supply to feed tightly regulated sub-1.0V core voltage at 10s of amperes to each FPGA.

This has been done on a smaller scale (maybe a few GB of memory) attached to a FPGA via PCI Express for acceleration of cloud applications, but I don't think anyone has taken it to the large amount of memory you want.

  • \$\begingroup\$ This is for a webserver cache application. PCI Express will be better than buying lots of individual servers connected through Ethernet. Microsecond latency is much better than millisecond. DDR4 is because it is the largest available module on the market, so with 8 modules I can fit 1TB on a PCI card. Ofcourse I will be powering this with external power, not from PCI bus. FPGAs are expensive, and why does it have to be Virtex? How much LUTs do you think this controller would use? I thought there were plenty of ASIC circuits for this, so I could avoid FPGAs. \$\endgroup\$ – Nulik Feb 13 '16 at 4:23
  • 1
    \$\begingroup\$ @Nulik - you say "PCI Express will be better than buying lots of individual servers connected through Ethernet. Microsecond latency is much better than millisecond." My estimate for latency of 10Gb ethernet, with 1kB packet is about one microsecond. So if latency is the driver, then your estimate is 1000x wrong. If latency between servers is the only obstacle to using 'commodity severs', then use off-the-shelf low-latency connectivity instead of Ethernet. The cluster-computing folks have this sorted. Or have shared-nothing cache and use the ethernet router to deliver requests to many servers. \$\endgroup\$ – gbulmer Feb 13 '16 at 11:52
  • \$\begingroup\$ the whole idea of this is to make the cost of connecting additional memory low. If I start buying 10 gig ethernet infrastructure that will be calculated in thousands of dollars and using additional servers the cost will exponentially elevate. Why can't I interface memory through the PCI express? The throughput will be 16Gb/sec more than enough for my app. I can connect not only 1TB but maybe 2 TBytes using 64 USD 8 Gigabyte DDR3 modules. This will lower the cost of 1TB of ram to 4,000 USD + the cost of PCI card, which I could make for under 500 USD. \$\endgroup\$ – Nulik Feb 13 '16 at 18:14
  • \$\begingroup\$ here are the measurements with networking, 15 microsecond latency for 32 byte payload if you tune the kernel parameters: blog.cloudflare.com/how-to-achieve-low-latency \$\endgroup\$ – Nulik Feb 13 '16 at 18:15
  • \$\begingroup\$ PCI express should give me 1 to 2 microseconds \$\endgroup\$ – Nulik Feb 13 '16 at 18:16

Not the answer you're looking for? Browse other questions tagged or ask your own question.