How does pipelined CPU access both code and data memory in real life?

Question

In the Digital Design and Computer Architecture RISC-V Edition page 442, a pipelined CPU is designed with separate instruction and data memories, as shown in the figure below.

However, this picture doesn’t show how modern CPUs work. Most of them use a single memory that holds both instructions and data. I’m interested in how this setup allows the CPU to fetch future instructions at the same time it reads and writes data. Also, are there different methods for embedded systems compared to general-purpose CPUs?

Feel free to reference reading material or anything I should look into.

Follow-up

Given that most discussions suggest the use of separate L1 caches for instructions and data, ultimately leading to shared memory, does this imply that the Instruction Fetch operation can take a variable number of cycles? For instance, it could take as few as two cycles in the following sequence:

1. Rising clock edge
2. Setting the inputs for the cache
3. Next rising clock edge
4. Reading data from the cache if it is present

Conversely, if a cache miss occurs and the cache needs to fetch data from main memory, could this operation take an arbitrarily larger number of cycles? Is that how things happen or is there a different approach.

Answer 1

Historically, there are two important basic architectures for processors: Harvard architecture and Von Neumann architecture. Harvard architecture has separate memories for instructions and data, so they can easily be accessed simultaneously, but loading and modifying programs is not directly possible. Von Neumann architecture shares the memory, permitting e.g. self-modifying code, but not permitting simultaneous access.

Modern systems combine these approaches in an attempt to get the best of both worlds. As mentioned by other answers, CPUs are often Harvard architecture at the low level, and separate first level I-cache and D-cache are used for instruction and data access. Beyond that the caches access the same memory system and the CPU will wait if a cache miss occurs.

_{(Image from AMD Ryzen 5 3600 Review)}

But as we can see in the diagram, modern systems do not use a simple pipelined CPU anymore. They use out-of-order execution which can run a queue of multiple instructions simultaneously and continue them when the needed data becomes available. They also have separate instruction prefetch units, which attempt to predict what execution path will be followed.

Simple pipelining processors are still used in embedded applications, such as microcontrollers. Many of these actually do have separate flash memory for code and RAM for data. Some architectures such as AVR and PIC go as far as loading data value from the flash memory requires special approach. More advanced ARM-based microcontrollers have unified memory space and are able to run from RAM, but still often run from flash directly.

_{(Image from ATmega328P datasheet)}

The issue of simultaneous access does have a fundamental consequence: CPUs with unified memory space usually have variable instruction timing, while Harvard architecture CPUs with separate memories take constant time per each instruction.

As you note, in the extreme case both the fetching of next instruction or loading a required data value could take a huge number of clock cycles. Fetching from main memory can pause the CPU for more than 100 clock cycles.

Answer 2

I'm not sure which version of the DD&CA you are using but if you check the next chapter you will see the cache architecture. It is also typical that code and data caches are separate, so the micro-architecture could really see separate memories for them, which are filled from the main memory or upper cache levels, even shared between cores and without distinction between code and data.

are there different methods for embedded systems compared to general-purpose CPUs? ... I am trying to close the gap between this model and the real world.

That's a huge gap to fill and there are many directions. If you are using the RISC-V edition of the book, you will find lots of open implementations to study.

Answer 3

There is one RAM for both code and data, yes; but that's not how the modern CPU accesses memory. A CPU will never read directly from RAM - it sends a request that takes data from RAM and pushes it into one of the (usually) on-die caches. Each cache-line is explicitly code or data.

For example, the Intel Core architecture has a separate 32 kiB instruction cache per-core (read 128 bits at a time). This feeds instructions into a 32B pre-decode fetch buffer (read six instructions at a time), which in turn goes into a 18 instruction queue. All those instructions are decoded and ordered before e.g. the ALU is involved to ensure best utilization. And this is also the reason why any mis-prediction or instruction memory write can cause serious performance issues - if you write to instruction memory that's currently in the process of being fetched, decoded or executed... the CPU has to throw all this out of the window and start again. Data memory read and writes also go through a separate (per-core) 32 kiB data cache.

Answer 4

However, this picture doesn’t show how modern CPUs work. Most of them use a single memory that holds both instructions and data.

They absolutely, certainly, do use separate data and instruction memories.

Those separate memories are filled from a shared memory and are called cache.

The modern desktop CPU does not execute straight from RAM. It executes from caches, and those have been separated for instructions and data for many decades - since at least 1970s. Even many "small" microcontrollers don't execute straight from primary memory, but have I and D caches.

I assume at somepoint memory will be spammed by cache reader

RAM is like a top fuel dragster. It can go very fast, but it takes a moment to get going. So, in most cases RAM can keep up.

But, it's not hard to come up with a bit of code that is indeed spamming the caches and introduces huge slow-downs. With a bit of creativity you can make any modern PC run at the speed of an ARM-based "Arduino" - precisely by "spamming" the main memory so that most accesses to the cache are misses. It requires some creative coding since you need to make branch prediction, prefetch, lookaside buffers, and a few other things not do their job.