Why is RAM not put on the CPU chip?

Question

Modern CPUs are very fast compared to all things external, including memory (RAM).

It is understandable, since CPU clock frequency has reached a point where it takes several clock ticks for an electric signal simply to run from from the CPU through the bus to RAM chips and back.

It also complicates life on many levels: multi-level cache hierarchies are built to deliver data closer to the CPU, which in turn require complex synchronization logic in the chip. Programs must be written in a cache-friendly way to avoid wait cycles while data is fetched.

Many of these problems could be avoided if a significant amount of RAM was located directly on the CPU chip. It doesn't have to an exclusive arrangement: maybe put 1-4 GB on the chip, depending on its class and allow additional memory installed separately.

I'm sure there are good reasons Intel, AMD and the like are not doing this. What are these reasons? Is it that there's no room to spare on the chip?

Answer 1

Intel's Haswell (or at least those products that incorporate the Iris Pro 5200 GPU) and IBM's POWER7 and POWER8 all include embedded DRAM, "eDRAM".

One important issue that has led eDRAM not to be common until recently is that the DRAM fabrication process is not inherently compatible with logic processes, so that extra steps must be included (which increase cost and decrease yield) when eDRAM is desired. So, there must be a compelling reason for wanting to incorporate it in order to offset this economic disadvantage. Alternatively, DRAM can be placed on a separate die that is manufactured independently of, but then integrated onto the same package as, the CPU. This provides most of the benefits of locality without the difficulties of manufacturing the two in a truly integrated way.

Another problem is that DRAM is not like SRAM in that it does not store its contents indefinitely while power is applied, and reading it also destroys the stored data, which must be written back afterwards. Hence, it has to be refreshed periodically and after every read. And, because a DRAM cell is based on a capacitor, charging or discharging it sufficiently that leakage will not corrupt its value before the next refresh takes some finite amount of time. This charging time is not required with SRAM, which is just a latch; consequently it can be clocked at the same rate as the CPU, whereas DRAM is limited to about 1 GHz while maintaining reasonable power consumption. This causes DRAM to have a higher inherent latency than SRAM, which makes it not worthwhile to use for all but the very largest caches, where the reduced miss rate will pay off. (Haswell and POWER8 are roughly contemporaneous and both incorporate up to 128MB of eDRAM, which is used as an L4 cache.)

Also, as far as latency is concerned, a large part of the difficulty is the physical distance signals must travel. Light can only travel 10 cm in the clock period of a 3 GHz CPU. Of course, signals do not travel in straight lines across the die and nor do they propagate at anything close to the speed of light due to the need for buffering and fan-out, which incur propagation delays. So, the maximum distance a memory can be away from a CPU in order to maintain 1 clock cycle of latency is a few centimetres at most, limiting the amount of memory that can be accommodated in the available area. Intel's Nehalem processor actually reduced the capacity of the L2 cache versus Penryn partly to improve its latency, which led to higher performance.* If we do not care so much about latency, then there is no reason to put the memory on-package, rather than further away where it is more convenient.

It should also be noted that the cache hit rate is very high for most workloads: well above 90% in almost all practical cases, and not uncommonly even above 99%. So, the benefit of including larger memories on-die is inherently limited to reducing the impact of this few percent of misses. Processors intended for the enterprise server market (such as POWER) typically have enormous caches and can profitably include eDRAM because it is useful to accommodate the large working sets of many enterprise workloads. Haswell has it to support the GPU, because textures are large and cannot be accommodated in cache. These are the use cases for eDRAM today, not typical desktop or HPC workloads, which are very well served by the typical cache hierarchies.

To address some issues raised in comments:

These eDRAM caches cannot be used in place of main memory because they are designed as L4 victim caches. This means that they are volatile and effectively content-addressable, so that data stored in them is not treated as residing in any specific location, and may be discarded at any time. These properties are difficult to reconcile with the requirement of RAM to be direct-mapped and persistent, but to change them would make the caches useless for their intended purpose. It is of course possible to embed memories of a more conventional design, as it is done in microcontrollers, but this is not justifiable for systems with large memories since low latency is not as beneficial in main memory as it is in a cache, so enlarging or adding a cache is a more worthwhile proposition.

As to the possibility of very large caches with capacity on the order of gigabytes, a cache is only required to be at most the size of the working set for the application. HPC applications may deal with terabyte datasets, but they have good temporal and spatial locality, and so their working sets typically are not very large. Applications with large working sets are e.g. databases and ERP software, but there is only a limited market for processors optimized for this sort of workload. Unless the software truly needs it, adding more cache provides very rapidly diminishing returns. Recently we have seen processors gain prefetch instructions, so caches are able to be used more efficiently: one can use these instructions to avoid misses caused by the unpredictability of memory access patterns, rather than the absolute size of the working set, which in most cases is still relatively small.

*The improvement in latency was not due only to the smaller physical size of the cache, but also because the associativity was reduced. There were significant changes to the entire cache hierarchy in Nehalem for several different reasons, not all of which were focused on improving performance. So, while this suffices as an example, it is not a complete account.

Answer 2

The main reasons larger memory (GB's of DRAM) isn't included on the CPU die itself is primarily about cost. CPU die space is significantly more expensive because of the manufacturing process required to make the very small features. It may also not be possible to manufacture the two on the same die, though I don't know enough about the details to give any definitive answer here.

Let's evaluate the feasibility of putting a large amount of DRAM directly onto the CPU die.

To give a comparison of scale, a modern CPU die might be ~180 mm\$^2\$ (approx. size of Intel Haswell dies). I don't have any accurate figures for CPU DRAM die sizes, but let's assume that 1GB of traditional DRAM takes 140mm\$^2\$ (calculated from GPU DRAM sizes). To a first order approximation, you are roughly doubling the CPU die size, which means at least doubling the cost of a CPU, and likely more just for 1GB of DRAM on the same die... I am not paying several hundred dollars just to get 1GB of DRAM, and I don't think anyone would.

However, the idea of sticking memory closer to the CPU is not completely a lost cause. This is likely where memory will move in the future because the fact is the speed of light is finite and it is only possible to communicate so fast over a certain distance.

Realistic techniques for moving memory closer to the CPU (note that these also have trade-offs with traditional techniques):

1. Stack them on top of the CPU itself. This is already done on the Raspberry Pi, and is part of the Wide I/O memory standard. The memory is still a separate die manufactured on a separate process. However, this has the problem that any heat dissipated in the CPU must pass through the memory before reaching a heat sink. This means it won't work for high power processors, and why the primary applications for this technology is in mobile processors/other embedded applications where your CPU isn't consuming many tens or hundreds of watts.

2. Stick them really close by on a lower-cost substrate. This is how HBM is designed to work, with a very large bus manufactured onto a lower-cost "interposer" die, and is the direction high-end GPU memory is going because the bandwidth is significantly higher. The memory chips and interposer are all still manufactured on different dies from the actual processor.

Answer 3

There are several reasons why adding large amounts of DRAM to a CPU could be infeasible.

1. The process and fab may not be set up for DRAM. DRAM requires special circuit elements that take extra manufacturing steps to produce. This increases the cost of manufacturing.

2. All that memory has to be tested. Memory testing increases your test time. That's another cost increase.

3. Enlarging the die is itself a cost increase, since it means fewer die per wafer. It also affects yield -- one defect takes out a larger fraction of your wafer. For an extreme example of this, look at the cost of full-frame (35mm) image sensors in cameras.

4. Developing a process that can handle special kinds of memory takes more time, money, and work, and has a higher risk of failure. Any problem with the DRAM would delay the release of the CPU. Desktop CPUs are on the cutting edge of semiconductor manufacturing, so a delay can be a huge competitive disadvantage. (See: AMD vs. Intel for the last several years.)

5. DRAM requires analog sensing for reads, as well as periodic refreshing. I'm not a DRAM expert, but I doubt that it could ever be as fast as a high-end CPU regardless of whether it's off-chip or on-chip. So you'll probably still be stuck with SRAM caching.

6. Even if you can overcome the above problems and cram a couple gigabytes of DRAM onto a CPU die, it's still not going to be enough to run a desktop PC, laptop, or server, so you'll have to have off-chip memory anyway.

Answer 4

In addition to the other answers, there's more that can be said about such a system. Moving memory to the main die would include a host of other engineering problems. You'd have to reroute buses, build a DMA controller into the main processor, restructure the IRQ bus, and determine how to get rid of all that extra heat you'd be putting out in a concentrated area. This means that the motherboard manufacturer would also have to get involved to support such a substantial change. While low-end systems would probably benefit from such a change, high-end systems would probably require substantial cooling. I doubt that the average laptop could handle such a chip, for example.

Such a chip would be vastly more expensive, although the main motherboard would drop in price (although probably not appreciably). If you've seen the packages for a DMA controller, plus the packages of RAM, you'd be hard-pressed to believe that all that logic could be pressed into a single die that wouldn't be substantially bigger. Also remember that CPUs are cut from big wafers that are a particular size. This means that the manufacturer would also have far fewer processors per wafer, which would increase the overall cost as well.

Keep in mind that we're not talking about wattage usage over the entire system, which would decrease, but rather that there'd be even more concentration of wattage (and thus heat) in a single area, which would probably increase the likelihood of failure.

Finally, there's another disadvantage here, and that is the ability to provide customized systems. Right now, manufacturers can choose to put out systems with identical processors but different amounts of memory, or different processors but the same amount of memory, based on the customer's preferences. To offer the multitude of different configurations, they'd have to build different dies, each on a different assembly line.

AMD is actively using technology that actually works that way, where each part of the processor die is a discrete logic unit that can be swapped out for different configurations. In the future, if such a design is viable, we could very well see CPUs that offer in-chip memory as an expensive module upgrade, perhaps swapping out some processing power in return, or other tweaks. For example, one day we might have the choice between 256 cores with no built-in memory, or 128 cores with built-in memory, or maybe even other configurations like part GPU, part CPU, part RAM.

Answer 5

Almost all of the above + one more additional problem: the heat.

The DRAM cells are essentially leaky capacitors. And the dielectric here is the SiO2 layer itself. As the temperature increases, the leakage currents increase proportionately. These discharge the DRAM cells far faster which would require much faster refresh rates, which would increase the complexity, required current and of course, add some more heat.

Answer 6

In addition to the answers already given there is one additional aspect: The waste due to production faults:

Let's say 1/100 of all CPUs of a certain model produced are faulty (in reality it is less, of course; 1/100 is easier to calculate) and 1/100 of all RAMs produced are faulty.

If both components would be combined on one single chip then 1/100 of all chips would have a defective CPU and 1/100 of all chips would have a defective RAM.

This would mean:

• 1 of 10000 chips would have both defective RAM and CPU
• 99 chips would have defective RAM
• 99 chips would have a defective CPU
• All in all 199 of 10000 parts produced would be waste

Producing separate chips the calculation is the following one:

• 50 of 5000 RAMs are defective
• 50 of 5000 CPUs are defective
• 100 of 10000 parts produced would be waste

Note that one GB of RAM is typically made in form of a bank consisting of eight chips so you would not have to combine two but 9 components into one chip if you want to put RAM and CPU on the same chip. This would lead to about 865 defective parts of 10000 produced in the simple example above.

The "IBM Cell" CPUs had exactly this problem. The "Playstation" console used chips that were partially defective; the Playstation software was written in a way that the defective cores and SRAMs were not used.

Answer 7

There are actually two types of RAM. They are static RAM and Dynamic RAM. Static memory are very fast but it comes in a higher cost. Meanwhile the dynamic RAM is slow compared to static RAM but comes in a cheap cost compared to static RAM.

Cache memory falls into static RAM. you can see that they comes in KB or MB sizes. They are fast. But high cost.

Answer 8

In addition to the other reasons mentioned, many systems have more than one CPU core. At times when information which is stored in main DRAM is consistent with all cached copies, all processors which do not have the information cached will have equal access to it. Some architectures are designed around the assumption that each CPU core will "own" a range of address space, and even if a CPU is capable of accessing memory which is owned by other processors, such accesses will be much slower than accesses to its own memory, but x86 is not generally implemented in such fashion.

If a system were designed around the assumption that processor cores own certain address ranges, and code should try to minimize use of other processors' address ranges, then it would make sense to have each processor code include a generous amount of on-chip memory. Such a design might reduce the time required for a processor core to access its own memory, but it would likely increase the time required for it to access another processor's memory. Unless the system is designed around such an assumption, however, it's likely that data would get distributed among processors without regard for who needed it when. Even if the extra time required to access data from within another CPU (vs an external memory system) were only half as great as the time saved on an internal access, 75% of the accesses in a four-CPU system would incur the penalty and only 25% would reap the benefit.

If one were designing a system from scratch to fit today's technologies, a non-uniform memory architecture could allow more "bang for the buck" than one which needs to allow all processors to access all memory efficiently. Given today's software designs, however, having an external memory system which is shared among processors is apt to be more efficient than trying to have bulk data storage within the processors themselves.

Answer 9

While all the previous answers are correct in pointing out why it's so difficult to add more memory to the CPU, it's also true that there is quite a lot of memory in modern CPUs.

In real-time operations when deterministic delays are important it is not unheard of to use the on-chip cache as addressable memory, for code and/or data. The advantage is fast and constant access time, and the disadvantage is that the on-chip memory is pretty limited.

Answer 10

The problems you describe in your original question could only be avoided if ALL of the memory in the machine were included on the CPU. Any additional memory added to the machine via slots on the main board would be subject to the same delays you describe and would require marshaling and logic control devices between the CPU / RAM and onboard RAM.

RAM is cheap and is typically expanded by users one or even twice between upgrading their CPU's.

Also remember that a typical "Fetch" call will, even if the RAM is running at CPU clock speed, lead to a number of idle ticks on the CPU.

The traditional organisation of memory on a machine is a pyramid with the CPU registers at the top, then cache, then RAM, then disc. Typically machines that perform well have a combination of a decent clock speed, a reasonable amount of cache, a good amount of RAM and a high speed hard disc (or array). In recent years, disc performance typically gives the best performance uplift in most PC's and high RPM discs, discs with cache and solid state discs give a good performance boost.

Yes, a machine with all of its RAM on chip would perform well in modern applications but a machine with some of its RAM on the chip running a standard operating system would likely not provide the performance increase that you may think it would.