What if a single transistor fails in a microprocessor or in RAM?

By failure I mean it completely stops working (due to overheating or some other reason) or gives incorrect output.

A single failure can change the specified microprocessor opcode to be read as some other instruction or mangle up the intended memory address in RAM and things can go downhill from there.

What safeguard/recovery strategy does a computer have against this?

closed as too broad by old_timer, Daniel Grillo, Bence Kaulics, laptop2d, Robherc KV5ROB Sep 2 '16 at 20:02

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • Before processes make it out into the "wild", they are around for years. 14nm was experimental 10 year ago and it's finally prime time due to consistent thresholds and not transistor failure. We are really good at making transistors at this point. – b degnan Sep 2 '16 at 12:50
  • I would expect during POST there is a kernel of self test executions for loopback on bus and compare with redundant Op Code. But only Intel and AMD don't will know what their self test fault detection levels are during power up. There are probably checksums for register operations and FSM codes used for self test in addition to illegal addressing range and of course, multi-core comparisons that may be used for self test. – Tony EE rocketscientist Sep 2 '16 at 12:52
  • For one non-I/O transistor to fail in operation when it previously worked is far, far less likely than for one (or likely several) to be bad or at least marginal from the start due to a manufacturing defect, so that is where most of the attention goes. Bombard the chip with energetic particles and things might start to be different. The places where you do see age failures of individual elements in ordinary gear are things like flash memory cells, but it's not really the transistor behavior that is failing, more the charge storage/non-storage behavior of the unique transistor structure. – Chris Stratton Sep 2 '16 at 14:51

In commercial consumer grade parts? None. It's tested at the factory, perhaps with a short "burn-in" process to guard against early failures. From then on any permanent failure will kill the device. Fortunately this doesn't usually happen for years if the device is kept within temperature parameters.

For military/aerospace systems, there are various sorts of redundant systems that can detect and recover from failures. Either by a process called "lockstep execution", or by simply having multiple computers like the Space Shuttle's famous five flight computers. (The Apollo project went to the moon with a single non-redundant computer!)

There is also ECC RAM, which is available for server-grade systems and can detect and correct single bit errors.

  • 4
    "From then on any permanent failure will kill the device" are you sure ? What if that single transistor failure is located in a non crucial CPU part like cache ? Wouldn't CPU survive, and some instructions simply give wrong results ? – tigrou Sep 2 '16 at 13:20
  • 2
    More common than transistor failure in space applications is the problem of cosmic ray interaction with silicon gates. "Radiation hardening" is a means of shielding a silicon device from harmful incident radiation. Even with this Rad Hard technology, multiple redundancy and majority consensus are used to produce highly reliable operation. – Wossname Sep 2 '16 at 13:21
  • 1
    @tigrou yes, I suppose it depends where the failure is; cache would probably fail POST, while things like the FDIV bug sat around for years before anyone noticed. It also depends on fail-open vs fail-closed - if you manage to short out an area of the chip the damage could spread. – pjc50 Sep 2 '16 at 14:43
  • 2
    @pjc50: Apollo had 3 redundant computers known as humans. Unlike the space shuttle, it could be flown if all its computers were down. My understanding is even during launch of the Saturn V, if the computers failed after engine startup the pilot could throw a switch and take control and have a reasonable shot of putting it in orbit and making the TLI burn. – Joshua Sep 2 '16 at 17:35
  • 1
    @tigrou - did you seriously say that the cache is a non-critical part? – iAdjunct Sep 2 '16 at 18:49

Way way too generic

The transistor could be in a section you don't use so you will never know.

It could be in a section you use, but the fail condition could still appear to work only ever needed a zero there, it failed to a zero and that is fine, I didn't notice it until perhaps a firmware upgrade that used that transistor differently.

Or it could be in a place that you use and the failure can result in any number of possible problems that may appear to manifest themselves in many ways. It is not possible to expand here certainly since we don't know what part you are using and how large the almost unmeasurable number of possible outcomes are a drop in the ocean doesn't begin to cover the number of hypotheticals. It is not worth talking about generally, even in aerospace.

For COTS systems, there is nothing whatsoever to protect you; you might have software that covers the dense or risky things like RAM, a POST memory test kind of thing. But if the failed transistor is as you suggest in the processor core, then you have a brick, again drop in the ocean, not possible to make a POST that covers the possibilities and reports on them properly, not worth the effort of even talking about it. You toss the thing and get another. Now when making the parts things like RAM that are higher risk might have alternate blocks on the die that can be fused in using the BIST or other screen tests. Likewise some products are derived from other products, to go back in the day a 80486SX vs 80486DX. If the floating point unit failed during chip manufacturing test, you blow the fuse(s) to turn it from a potential DX into an SX and sell it that way.

Chip screen looks for and, with experience, covers more than say 99.99% of failures. Depending on the product there is expected to be one in some number of thousands or tens of thousands that fail down the road. That is just how it is.

Aerospace, radiation-hard in particular, flip flops are or were when you still had folks around who know how to build reliable stuff (they have mostly retired now) are triple voted, three sets of transistors per, if one bit fails or experiences a single event upset (which is the primary reason for the triple voting) then the other two dominate the vote. You could argue that you get the what-if-one-fails argument. But you don't triple vote everything combinational logic isn't; if it gets a hit it should settle out before latching into the next flip flop/bit. You also need much beefier transistors to deal with single event upset, and you also have more other material to deal with total dose to extend the life of the part before the whole thing dies from exposure. Decades of experience (all lost to the young-uns who grew up in the disposable age, make space trash instead of make one that always works). A single transistor failing is not a primary concern, single event upset, and latch-up are (which could/would lead to destruction if not handled) total dose, etc. Single point of failure along with MTBF are very important; the MTBF should be longer than the mission life, and no single point of failure should be able to go without detection and resolution. Two failures at once are not normally solvable (drop in the ocean).

With COTS you don't worry about a single transistor failing; it is all about averages/statistics. You start with experience and there are not that many foundries anyway and they know what they are doing. You follow the decades worth of experience in design, layout, design verification, testing, etc. And you most definitely expect a yield less than 100%, and then of the parts packaged and delivered you expect some percentage of those to fail in the field. If your customers are expecting COTS quality and pricing, but fault tolerant or military or aerospace grade parts, you just need to educate the customer that what they are asking for is an order of magnitude more in price and an order of magnitude slower in bandwidth/processing power. As well as an order of magnitude longer lead time.

RAM being dense is an area focused on a lot, not for a single transistor failure, but more for single event upset, one bit flipped. So ECC/EDAC are used, as mentioned in manufacturing there might be a BIST and an alternate bank that can be fused in. With things like flash and/or hard drives they are failure prone so beyond the view of the user there may be extra banks/sectors that can be marked bad and others swapped in. It is possible with RAM, but normally not as you don't use RAM the way you use non volatile storage, instead you use DIMMs for example and have the user replace them if/when they fail. Normally like the processor socket that is for upgrades or configuration choices, not because RAM fails at any great rate compared to other components on the board.

This is a statistics game. Any one transistor can have a critical effect (maybe its in the program counter incrementor), an effect on performance (say a cache line never hits), or a user-experience degradation (stuck pixel or similar). Or maybe the failure is in test logic, so its invisible to normal operation.

For some applications, these random failures just contribute to the device lifetime (battery failure, or percussive impact being more significant risks). Transient faults are just as important as permanent hardware faults.

Server systems need to be able to detect end-of-life for individual nodes, safety critical systems are more likely to employ detection and redundancy - however observing the less terminal failures is generally hard. Contrast this to a magnetic HDD where error correction rates (and other metrics) can often be measured prior to the point of catastrophic failure.

Exhaustive self-test is time consuming - see http://www.memtest86.com/ for an example of a simpler problem than testing a processor.

Not the answer you're looking for? Browse other questions tagged or ask your own question.