How to find and overcome run-time RAM corruption in a Microcontrollers? [closed]

Question

How can I find and overcome any RAM corruption in a Microcontroller (ARM Cortex M0) during run-time? For example what if two or three locations are corrupted, say 0x2E 0x2F,0x30. How can I still let the system run by overcoming or ignoring this corruption.

Answer 1

You need to pick the right MCU which has the features you need. Here's an example of Cortex M0 chip which has both EDAC (correction of single-bit errors) and Scrub (periodical refresh of memory contents which prevents error accumulation).

If you don't have error correction hardware features, the best you can do is periodic check of memory content which wasn't expected to change and a reboot if a change was detected.

Answer 2

How can I still let the system run by overcoming or ignoring this corruption.

On a general Cortex M0: probably not at all. Whilst this becomes easier (not easy) if you have an MMU that can map memory around, your ARM microprocessor very likely doesn't have that. So you'd have to write your software and linker scripts in a way that specifically avoid these addresses, and compile that, and run that on the µC.

How can I find ...

Typically, that's the job of memory-validation hardware mechanisms. In the world of PC/Server/Workstation/Cluster-style computation, you'd go for ECC memory and possibly even memory redundancy. In spaceborne applications, you'd possibly have memory controllers that have much more elaborate Forward Error Correction (FEC) on the RAM – the same kind of thinking that solid-state mass storage uses to avoid bit errors, or your digital TV receiver.

The problem here is: you don't have a memory controller that can do that on a Cortex M0. You could of course first check every RAM region against a checksum before accessing it often in software, but that'll be an enormous task, and also won't really help if the memory corruption happens during repeated reading or a write.

So, if a Cortex M0 has broken memory, it's broken and needs to be replaced.

Good thing is that this kind of chip definitely doesn't have that much RAM, so the bit error probability due to the Bernoulli distribution caused by "or"ing a lot of "is this bit broken?" probabilities isn't that high. Which all the more means you should replace the chip in the field if it has broken RAM – there's something wrong with the chip, and it's hard to predict what'll fail next.

Answer 3

Cortex M0 does not have any specific fault detection or reliability features such as ECC. There are microcontrolers (Cortex-M7 based for example) which have some of these features, these mught help you improve reliability.

Leaving aside the cause of the errors (a high radiation operating environment would be a reasonable case), you need to achieve at least two things. Detection of the fault before too much damage has been done to your saved state, and recovery of the system. These are non-trivial, and the best approach does depend on your application. Things you can consider are:

• Watchdog
• Executable checksums
• Data region duplication and checksums
• Replicated threads or cores
• Data validation (repeatedly)

Once a fault is detected, the most sensible thing to do is reset the core and start again. Simply resetting the core with an external timer every so often might help.

Answer 4

RAM being overwritten can be a symptom of a software bug rather than a hardware failure or fault. The two most common causes in embedded C programs are due to pointers and out of bounds array accesses. Often debugging tools allow you to set breakpoints on memory writes to specific locations, so you might be able to find the line of C code that is responsible for the overwrite and then determine why this bug occurred.

Even without such support, you might be able to narrow down the suspect lines by finding out what variables or array have been allocated to just before these addresses, since that provides a hint. For example, if there is an 8 element array, and each element is 1 byte, and it starts at 0x2E-0x08=0x26 then if some code is writing to this array but the index is 9, it would overwrite 0x2E. (Maybe you have an array of structures, each of which is 3 bytes, so the "bad data" could give you a clue as to where this is coming from).

However, if it is a case of an uninitalised pointer that is being used, the memory layout is unlikely to be any help. But you can still search your code for pointer usage or use static code checkers, or your compiler warning levels to look for pointers being used before being initialised with a valid address.

Finally, if you can repeatably produce this overwrite but can't determine which line of code is the cause, a brute force approach is to remove sections of code that you suspect until you find it. However, this risks disturbing the rest of the code so much that the bug moves.

In the case of a software bug, you do not want to find a way to let the system still run with such corruption; you should find and fix the bug. If you were to set linker options to avoid this area of memory, the bug would likely move elsewhere (provided you don't have defective hardware).

Answer 5

where high reliability is needed often several systems are run in parallel and they vote on the decisions.

Answer 6

In the end, you simply can't protect yourself against all memory failures. Fortunately, these are extremely rare unless your device goes into space.

You can, using extra cycles, protect yourself from corruption in certain specific parts of memory. For example, you could keep multiple copies of a critical data structure, each with its own checksum. You'd have to access data in this section thru subroutines that update the multiple copies, compute the checksums, and deal with discrepancies.

Many of the things you hear out there as strategies, particularly for dealing with program memory failures, actually make things worse. For example, if your code takes 1 kB, but with error checking takes 2 kB, you have just doubled the chance of getting hit by a random error. Whatever checking you do now has to overcome that just to break even. And how do you check the checker? As I said in the beginning, you simply can't protect yourself against all failures.

A more realistic strategy is to minimize the chance of failure in the first place. You could use a old microcontroller with a larger feature size. That makes it less likely certain events can flip a bit. External memory subsystems can have error correction and detection, up to some level of errors.

If something flips a bit in the ALU or one of the key registers of the processor, you're going to be screwed. In some cases a watchdog can help. When the code no longer clears the watchdog timer regularly the processor is reset and you get to start over.

Basically, if you need extra high reliability, it will be better achieved with multiple redundant systems at a higher level than with cutesey tricks in a microcontroller.

Answer 7

You can use error correction coding (ECC) in order to correct or detect memory errors. This will significantly increase the size of the RAM utilization, and introduce processing overheads.

ECC works by using only a set of the available words. As an example set 000, and 111 as valid words. If you read 101, you know this is not a valid word. ECC is a broad subject with many different schemes of various complexity. The above scheme, for example, is called a repetition code.

Although the probability of getting an error increases, due to the increased memory usage, your ability to correct the errors increase faster. The question becomes what bit error rate (BER) you can tolerate while keeping within your other constraints. You can keep increasing the code word in order to achieve your desired BER, sacrificing memory efficiency.