# Is there a correspondence between cache size and access latency?

Is there is a correspondence between cache sizes and access latency? All other things being equal, does a larger cache operate slower? If so, why? How much slower?

-
If you are interested in exploring some trade-offs in cache design, HP developed the tool CACTI which provides estimates for latency and power use based on design characteristics. A web interface for version 5.3 is also available. – Paul A. Clayton Sep 21 '13 at 15:22

Items in your hands are quicker to access than items in your pockets, which are quicker to access than items in your cupboard, which are quicker to access than items at Digikey. Each successive type of storage I have listed is larger but slower than the previous.

So, let's have the best of both worlds, let's make your hands as big as a Digikey warehouse! No, it doesn't work, because now they aren't really hands any more. They're a cannonball weighing your down.

The reason larger storage is slower to access is distance. Larger storage is further away from you on average. This is true for physical items, and for RAM.

Computer memory takes up physical space. For that reason, larger memories are physically larger, and some locations in that memory are going to be physically further away. Things that are far away take longer to access, due to whatever speed limits there are. In the case of your pockets, and Digikey, the speed limits are the speed of your arms, and the highway speed limits.

In the case of RAM, the speed limits are the propagation speed of electrical signals, propagation delay of gates and drivers, and the common use of synchronous clocks. Even if money was no object, and you could buy as much as you want of the fastest RAM technology available today, you wouldn't be able to benefit from all of it. Lay out an A4 sized sheet of L1 cache if you like, and place your CPU right in the centre. When the CPU wants to access some memory right in the corner of the memory, it'll literally take a nanosecond for the request to get there, and a nanosecond for it to get back. And that's not including all of the propagation delays through and gates and drivers. That's going to seriously slow down your 3GHz CPU.

Since synchronous logic is a lot easier to design than asynchronous logic, one 'block' of RAM will be clocked with the same clock. If want to make the whole memory an L1 cache, then you'd have to clock the whole lot with a slow clock to cope with the worst case timing of the most distant location in memory. This means that distant memory locations are now holding back local ones, which could have been clocked faster. So, the best thing to do would be to zone the memory. The closest and smallest section of the cache would use the fastest clock. The next closest and smallest section would use a slightly slower clock, etc.

And now you have L1 & L2 caches and RAM.

Which brings us to the next reason, power consumption.

The cache actually consumes a significant amount of power. Not only the memory itself, but all the logic surrounding it which handles the mapping between the cache lines and the main memory. Increasing the performance of this extra logic can result in an increase in power consumption. Now, for certain applications (mobile, embedded) you have even more incentive to keep the cache small.

See Cache Design Trade-offs for Power and Performance Optimization: A Case Study (Ching-Long Su and Alvin M. Despain, 1995).

-
+1 I like your answer. Hits all the relevant points and backs it up as well. Easy to read – Gustavo Litovsky Sep 20 '13 at 1:25
is that right? answer is premised on the delay in speed of light of an inch versus several inches? – Andyz Smith Sep 20 '13 at 4:58
This answer is good as a general overview, but I suspect it does not answer the initial question. One shouldn't be a uArch expert in order to realize that A4 sized cache is not practical. I believe OP asked about some reasonable increase in cache's size and how this increase will affect the latency of the cache access. – Vasiliy Sep 20 '13 at 8:35
@AndyzSmith - It's a gross exaggeration and over-simplification, but I think it gets the general point across, that at some point, that L1 cache can't be enlarged indefinitely. At some point, the laws of physics come in, and slow it down. – Rocketmagnet Sep 20 '13 at 9:36
The delay of an on-chip signal is not the speed of light, it's a more complicated value computed by considering the net/wire plus driven gate capacitance as an RLC network. It's not long enough to be a transmission line. The longer the net, the larger the driving gate needs to be, or you need to insert buffers as repeaters. All of which also consumes more power. – pjc50 Sep 20 '13 at 9:59

Leaving out all the economical/performance/power consumption factors, the answer to your question is: it depends on many micro architectural factors.

As an example see this reference - the measured L1 access latency for all the processors under test is 4 clock cycles. The frequencies of the processors are almost the same, but the sizes of L1 cache differ by up to a factor of 3.

The reason for the constant latency to L1 across several different processors in the above test is rooted in the micro-architecture of the cache: the cache access itself (retrieving data from the memory) takes just a single clock cycle. Additional three cycles are spent for decoding the access, checking for data presence and more... The number of the additional stages is the same across the processors in test, therefore the access latencies are the same.

Despite the above example, one should not come to conclusion that cache latency is independent of cache's size. If someone would try to implement a ridiculously large L1 cache, the logic which performs all the required operations for a cache read would also become large. At some point, the propagation delay through all this logic would be too long and the operations which had taken just a single clock cycle beforehand would have to be split into several clock cycles. This will rise the latency.

Assuming that the cache in question implemented in SRAM, the modules which affected by cache size the most are: row decoders and muxes. However, even sense amps will be affected for very large caches: smaller voltage swing on a bit line due to higher capacitance will require a "stronger" sense amp. Said that, the most severe effect on logic speed will be added by wire interconnects capacitance - this capacitance has more than a linear dependence on the SRAM size. The exact details are implementation specific.

Now, L1 caches are pretty stable in their latencies because their performance is the most crucial. If you try to analyze L2 and L3 caches, the picture complicates a lot.

The picture complicates much more when you consider multi-core processors - they have additional logic for ensuring cache-coherency. This leads to an additional factor which affects the latency of cache access: the history of accesses to memory of all cores.

Summary

As you can see your question is far from being trivial and can't be answered completely. However, if you consider economically and performance preferable caches, then I'd say that their size will not affect the latency in any appreciable way.

This reference is a very in-deep analysis of modern CPUs' performance factors. There is a lot of cache related material in there. Requires deep understanding in computer architecture and micro-architecture principles (alternatively - a good summary of topics one need to know in order to become a professional in this field).

-
Thank you for the answer! Yours and @Rocketmagnet's answers are pretty much complementary. Hope I could choose both. I've already got my copy of the reference cited and been very interested in the topic lately, hence the question. – ivanmp Sep 20 '13 at 9:48
why does the logic to perform a read operation depend on the cache size.. Why does it 'become too long' at what specific number of address is there an discrete jump. or if no discrete jump, what is the formula for access time versus size? – Andyz Smith Sep 20 '13 at 13:03
can you be specific about which component: "The critical recurrence, then, is an adder, a decoder, the SRAM word line, the SRAM bit line(s), the sense amp(s), the byte steering muxes, and the bypass muxes." causes the logic to become large? en.wikipedia.org/wiki/Sum_addressed_decoder – Andyz Smith Sep 20 '13 at 13:35
@AndyzSmith, the formula of access time vs size can only be given by someone designing and simulating the cache. In the reference I posted you can see it takes 4 clock cycles to get a data from L1, but no one attempts to estimate the slacks associated with these reads. Asking for a formula is way not practical question without a lot of additional implementation specific information. – Vasiliy Sep 20 '13 at 19:53
@AndyzSmith, assuming that the cache in question implemented in SRAM, the modules which affected by cache size the most are: row decoders and muxes. However, even sense amps will be affected for very large caches: smaller voltage swing on a bit line due to higher capacitance -> "stronger" sense amp. Said that, the most severe effect on logic speed will be added by wire interconnects capacitance - this capacitance has more than a linear dependence on the SRAM size. Again, the details are implementation specific. – Vasiliy Sep 20 '13 at 19:56

CPU cache test engineer here - Dave Tweed in the comments has the correct explanations. The cache is sized to maximize performance at the CPU's expected price point. The cache is generally the largest consumer of die space and so its size makes a big economic (and performance) difference.

Take a look at Intel's Ivy Bridge CPU family page: http://ark.intel.com/products/codename/29902/Ivy-Bridge

The top end Xeon comes with 30MB of cache, has 12 cores and costs about $2700. The lower end i3 with 3MB of cache (i3-3217) costs just$500 for a whole laptop (I can't find it individually).

The Xeon gives the ultimate performance but it also costs more to manufacture and test. The i3 is much cheaper but the trade-off is a smaller die size of which the cache is the largest part.

-
Very interesting answer. But as I stated in the question, I'm not looking for the economical, obvious, answer. What I'm trying to understand relates to the performance involved in accessing a very (>512 MB) large cache. Would the size degrade the cache's performance? – ivanmp Sep 19 '13 at 16:48