What stops an assembly program from crashing the operating system? [closed]

First of all, I am a beginner, so if this question sounds silly, please point out the incorrect assumptions.

From what I understand, the job of an operating system is to manage hardware and the software that runs on the OS. Also, from what I understand, assembly programs allow one to control hardware almost directly. In an assembly program, one can read and write data into registers, and read and write data into RAM.

Given this freedom to mess with registers and RAM, would it not be possible for assembly programs to affect the operating system? Suppose an OS is using register A to store crucial information, and suppose I run an assembled program on that OS. If the program successfully writes junk into register A, the OS will surely be affected.

Questions:

1. Is it possible to mess with register A in the manner described above?

2. If not, what stops assembly programs from modifying the registers used by the OS?

closed as too broad by The Photon, Dmitry Grigoryev, Nick Alexeev♦Mar 11 at 18:09

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

• smart programmers... – Sunnyskyguy EE75 Mar 11 at 4:22
• There are a lot of computer architectures now and were in the past, as well as many operation systems were developed. What architecture/OS are you referring exactly? On some (old) architectures there were no possibility to stop program from what it was doing after start, that's correct. But modern hardware/OS have built in hardware tools which give only the part of the memory for the program in "normal" mode (not superuser), and it can't access memory outside this limit. Registers are free to use, as OS don't store any useful info in registers, just in memory/on disk. – cyclone125 Mar 11 at 4:24
• In a microprocessor your program runs in "user mode", the operating systems runs in "system mode." If a user mode program executed a halt instruction, for example, the machine would not halt. The halt would be trapped and the operating system invoked. Regarding RAM, the OS would set up an environment for the user mode program so that, via memory management hardware, what the user program sees as RAM address X would not really be RAM address X. – George White Mar 11 at 4:26
• @flux I have updated my comment. The answer would be: there is no "general" answer to your question, as there are/was different computer architectures/OSs. It can be in different ways. – cyclone125 Mar 11 at 4:33
• ... Long ago I used to write in raw machine code. Yee ha !!! :-) – Russell McMahon Mar 11 at 4:58

In the end, all programs are machine code, regardless of whether the source language was assembler or a high-level language.

The important thing is that there are hardware mechanisms that limit what a given process can do, including "messing with" registers that could affect other programs or the operating system itself.

This started with a simple distinction between "user" and "supervisor" modes of operation, and has since evolved into a security architecture with multiple "rings" of privilege.

Here's a very generic example to make it a little more concrete:

• In "user mode", a process cannot access memory that hasn't been assigned to its process ID. Memory assigned to other processes and the operating system itself is blocked. This includes the values of registers used by those other processes. This is enforced by the MMU hardware.

• Therefore, in "user mode", a process cannot access the MMU control registers.

• And obviously, in "user mode", a process cannot change the mode to "supervisor mode" except through a very well-defined mechanism that involves calling an operating system function.

The operating system gets control if the process tries to break any of these rules. Once that happens, the OS can simply halt the offending process, never executing any more of its instructions.

• If I understand correctly, what you're saying is: Some processors have "user mode" and "supervisor mode". The OS runs in "supervisor mode", and puts the processor into "user mode" to run my fictional assembly program. In "user mode", there are registers and RAM addresses that the assembly program cannot access due to deliberate design of the hardware. – Flux Mar 11 at 4:52
• Basically, this answer describes modern "i386-like" architectures with MMU and Protected mode. But to be true there are many old (i8080, MOS 6502 etc.) as well as modern simpler (AVR, ARM Cortex-M etc.) architectures that don't have these features and if some kind of OS is used (e.g. old CP/M, modern FreeRTOS, ChibiOS etc.) nothing can stop the program from what it is doing. – cyclone125 Mar 11 at 4:52
• @Flux The i386 (and above) architectures provide details to learn. The x86 architecture not only has memory addresses that can be protected, but also I/O addresses, too. (There are different instructions used for memory access vs I/O access.) There are three memory addresses in the i386+. The segmented/selector-based address uses a selector register that references a table entry (GDT or LDT) to map the pair into a single 32-bit linear address. Paging tables then translate the 32-bit linear address into a 36-bit physical address (P-II.) Protection exists at both translation steps. – jonk Mar 11 at 5:03
• @flux your summary is correct. A program in a protected memory system with a suitable multitasking OS should not be able to crash the system with any stream of instructions. Even invalid ones - those end up at a special handler. – pjc50 Mar 11 at 5:27
• Even though there are multiple rings (in x86 at least), it's very, very rare for any more than two to actually be used. – forest Mar 11 at 11:00

Many multitasking operating systems use a data structure called a Process Control Block (PCB) to take care of the register overwrite problem. When you run your code, the OS creates a new process to keep track of it. The PCB contains information about your process and space allocated to hold register contents. Let's say process A is currently running on the processor and your code is in process B. What happens when you run your code goes something lke this:

1. Process A's state data (register contents, program counter, etc.) are copied into its PCB.

2. Process B's state data are copied from its PCB into the CPU registers

3. Process B runs on the processor until it finishes or is preempted

4. Process B's state data are copied back into its PCB

5. Process A's state data are copied back into the CPU and it continues running

If the operating system is running as process A, then you can see how saving its registers before your program runs then copying them back into the CPU once your program ends prevents user programs from messing with what's going in other processes.

To avoid user processes writing over OS data in memory, most platforms use memory segmentation. Basically, using virtual memory, the address space a process sees can be mapped to any arbitrary range of physical addresses. As long as the processes' physical memory spaces don't overlap, it's impossible for one process to overwrite another's data.

Of course, different OSes do things differently, so this won't apply in every case. Also, on most platforms, OS processes run in a different mode from user processes and there are some intricacies there.

Depends on what platform you're talking about.

• On a more basic processor, the processor just executes whatever instructions the program tells it to execute.

• On a more sophisticated processor, there's (at least) two modes. In one mode, the processor does whatever the program tells it to do. In the other mode, the processor itself refuses to execute certain instructions.

What stops a program crashing the whole system? With the first type of processor, the answer is "nothing". With a basic processor, a single rogue program can indeed crash the entire system. All the early 8-bit home computers, and many of the 16-bit ones, fall into this category.

On a modern PC, the processor has "protection" hardware. Basically the OS runs in a special mode that lets it do anything, whereas normal programs run in a mode where the processor will only allow certain actions (based on what settings the OS has configured on the processor). Stuff like only accessing certain registers, or only accessing particular memory ranges.

Obviously, allowing a single rogue program to crash the entire system is bad. (There are also severe security implications in letting a rogue program access whatever data it wants.) To avoid this, you need two things:

1. A processor that actually has protection hardware (i.e., it can be configured to refuse to execute certain instructions).

2. An OS that actually uses these facilities to protect itself. (It's no good having a chip with protection circuitry if the OS never uses it!)

Just about any modern desktop OS you can name (Windows, Linux, Mac OS, BSD...) it a protected-mode OS running on a processor with protection hardware. If you're doing embedded development on some 8-bit microcontroller, that probably doesn't have any protection hardware. (Or any OS, for that matter...)

Q. What stops an assembly program from crashing the operating system?

A. Nothing.

However, a lot of very clever programmers have tried very hard over the years to make it more and more difficult. Unfortunately, for every clever programmer, there are many, many others who between them are more creative, more ambitious, and sometimes just luckier then the clever ones. Every single time a clever programmer says that nobody should, would or could do something, someone out there will find a way to do it. Microsoft Windows (as an example) has been around for almost 35 years and we still have BSoD (Blue Screens of Death), which are just instructions that crashed the operating system.

Lets start with a little terminology. Everything that runs on a computer does so in machine code. The bit that reads the keystrokes or the movement of the mouse pointer, the bit that changes the colour of a pixel on the display or reads a byte from a file and the bit that calculates whether your bullet hit the bad guy or the bit that decides if your credit card application will be accepted are all executed as a sequence of machine code instructions. Some jobs are so common and are done so often that it is makes sense to assemble the instructions required to do them and have everybody use that assembly. The bunch of these jobs that allow or help others to use the computer tend to be called the operating system but there is nothing inherently different between them and any other programs. They are all just sequences of machine code instructions. You will hear about protected modes and privilege rings but these are just more sequences of the same machine code telling the computer how to behave.

What makes operating systems more complicated (and therefore prone to crashing) is that they have to account for things that you don't normally have to think about. Take the simplest of jobs as an example. I want to write a message to the end of a file. In a high level language you would write something like:

  with open("myFile.txt", "w+") as f:
# do some really clever things
f.write("Goodbye cruel world!")


Lets ignore all the details about how the physical states are accessed and changed or how they are interpreted as bits and bytes or how those bytes are transferred to and from the memory and CPU, and trust that all that is handled by programs that the OS provides behind the scenes. Lets just think about the how you append to the end of a file. 1) Find out where the end of the file is, 2) write something at that position. What could possibly go wrong? Actually, quite a lot. Think about what else is happening on the computer while you are doing clever stuff. If anything else being done by anybody else (including the operating system itself) changes the file you are working on in any way, then this really simple job suddenly gets a whole lot more complicated. The file is longer, the file is shorter. The file is not there anymore. The disk is full, the disk would be full if all the bytes are written, the disk is locked, the disk is faulty... There are now so many different conditions and exceptions that the operating system has to deal with that it is amazing it ever gets anything done at all.