Historical Addressing Modes/Conditions that Complicate Page Fault Handling

Question

I was reading this linked question and the following paragraphs stood out to me:

For example x86 and 68000 had instructions which access memory two or more times. Each memory access could cause a page fault. Hence the CPU must either roll-back the incomplete instruction and save enough state to re-run the instruction from the start, or save enough state to pick up and continue the incomplete instruction. In either case, that might be millions or even billions of instructions later, with other instructions in between also suffering page faults.

...So 'incomplete instruction exceptions' save different state on the stack, and hence have to be handled differently for normal interrupts.

Today, CPU architectures except for x86 by and large don't permit multiple memory accesses per instruction. That said, I am wondering historically what addressing modes, and CPU state (such as a page fault in the middle of an instruction) would require special state to be saved to restore processor state? Additionally, what CPU state would conceptually need to be saved?

For the remainder of this question, assume I have a CPU that restarts an instruction on page fault (x86), and one that stops in the middle of an instruction on page fault (68k).

Also assume these CPUs are is a in-order CISC-like; they allow both src and dst to be memory operands for MOVing data and arithmetic operations can access memory directly. This creates an ISA where insns can have multiple memory accesses. Additionally, ignore complications caused by pipelining. I'll ask about x86's edge cases, as alluded to, as a separate question.

I can think of only two badly-behaved historical addressing modes offhand- pre decrement and post-increment. Additionally, an instruction that spans a page boundary can cause issues.

• Pre-decrement/Post-increment
  • On a "restart on page fault" CPU, state for a faulting pre-decrement would need to be able to revert the dec/incremented register, regardless of why the instruction faulted (span page boundary or page fault on access); decrement would occur before any memory access, but increment will occur between memory access on MOV.

• Instruction spans pages

  • On "restart on page fault" CPU, an instruction may have started already while waiting for the remaining instruction bytes. So the instruction boundary needs to be known to refetch (as well as state to revert side effects of the insn's addressing mode) after the page fault handler runs.

• AFAICT, the only state that needs to be saved on a CPU that "stops in the middle of an insn on a fault" is "how far the instruction proceeded" before the fault, and continue from that point, regardless of fault reason. Registers need need not be reverted, and I don't even think the "fault reason" needs to be saved.

Is my explanation/understanding accurate? Are there any other CPU state and addressing modes (plus their combinations) that I should consider?

Answer 1

The Motorola series of CPUs stored different amounts of processor-specific state on the supervisor stack when handling most faults (including page faults). This (intended to be opaque) binary blob was, in essence, a dump of the CPU's internal state at the time the fault was discovered, allowing the CPU to pick up where it left off should you decide to restart the operation. However, I'm not aware of any other processor that (re)stored its internal state in this way.

Disclaimer: speculation based on vague memories follow. While storing microarchitecture state on the supervisor stack was effective, I recall that it also impacted forward/upward compatibility of operating systems (this was totally hidden from user-mode). The rough inability to predict how changes in the microarchitecture would impact the fault handling logic made supporting kernels more difficult than, say, for x86 where backward compatibility was a philosophy set in stone for not just ring-3 code, but for all rings.

Another concern is latency; Motorola's approach to solving this problem involved automatically storing up to (IIRC) 16 long words to or from the stack. You don't get a say over this, which makes Motorola's processors some of the slowest-to-respond to various kinds of faults. Since different operating conditions can affect which stack frame gets pushed, latency further wasn't predictable without very detailed knowledge of internal CPU state at the time the fault occurred. One good reason why processors elect to restart from scratch instead of storing current state is that it actually improves performance and makes things more predictable.