What determines the maximum size of a cpu cache?

Question

Looking at a list of the very latest CPUs, I see several of them with a cache size of 12MB or 8MB - pretty small, when compared to the ever-increasing size of hard drives and ram. It seems to be taken for granted that a cpu cache will always be small, but, why is that? Is it just economicaly unfeasible, or are there engineering reasons why it has to stay small?

I'm thinking of some cryptocurrencies (i.e. ethereum), that are designed to be memory-hard, so the speed of the algorithm is limited by the IO bandwidth of the memory, with the idea that this makes it impossible to design a custom chip specifically to solve that algorithm as was done for bitcoin. But, if someone was making a custom chip anyway, couldn't they just cram a gigabyte into the cache and do away with the IO bottleneck?

Answer 1

It's a trade-off between the higher hit rate of a large cache and the faster speed of the smaller cache RAM. Hit rate follows a law of diminishing returns as the cache gets larger.

Doubling the size of a large cache may only less than a couple of percent increase in hit rate but it'll certainly increase its access time, which slows CPU throughput.

Answer 2

The issue is structure size vs. signal propagation speed.

If you build a larger cache, it takes up more space physically, meaning the length that the signals have to travel increases, which reduces the maximum clock rate the cache can be run at. The L1 cache needs to run synchronous to the CPU in order to be useful, so the size of the cache is a limiting factor in clock speed.

Other cache layers can be larger and run at slower clocks, but this requires the CPU to wait until the cache answers.

The GPU approach is to have lots of threads per core (i.e. a SMT factor of 16 or larger, compared with 2 on Intel CPUs or 4 on POWER). This means that each individual thread will only run at a fraction of the clock speed, but results for a memory access are not required until all the other threads have begun their memory access, at which point the result for the first access should be ready.

This is why GPU mining is interesting for these coins.

Answer 3

Gigabyte-sized memory (DRAM) don't use the same manufacturing process as CPUs (logic), so you can't have both within the same chip, unless you are willing to make compromises that would make the whole thing inefficient.

Cache memory uses static RAM, which is more alike logic, but which takes more silicon space than DRAM, so you can't have as much, practically. Moreover, big cache memories uses content-adressable cells, which is even more complicated to make. In short, making a gigabyte-sized cache is not really doable economically.

But anyway, even if you were able to do it, it wouldn't solve your problem. You would maybe gain a few cycles of latency, but would still need to have a bus between the processor and the memory, even within the chip. So that would be your bottleneck in the same way it is on a regular computer.