Silicon bugs, errata sheets

Question

In many (most??, all??) microcontrollers that I have used over the past years, there where sometimes some silicon level bugs, and the manufacturers provide the engineers with the errata sheets, describing what unexpected behaviour they may face.

Why don't they ever fix these "bugs"? Since the product is still produced, and in most cases solving the problem won't affect the previous implementations, why they do not just revise it? In many cases the product may be stabilized, most bugs may have been found, and may have a significant part of its product life-time ahead of it.

Is it so difficult (technically)? Expensive?

Answer 1

Critical bugs do get fixed. Usually they're fixed before the product enters production. Unless you're using early samples, you might never see the worst bugs.

Fixing bugs is difficult and expensive. It's not just changing one line of RTL code. If you did that, you'd have to resynthesize, redo the physical layout, tweak the layout to fix any timing problems, buy a whole new mask set, produce new wafers, test the wafers (normally), validate the new fixes, and possibly characterize or qualify the product again. This takes months and costs a distressing amount of money. For that reason, we try to fix bugs directly in the layout (preferably on a single metal layer). This is faster and cheaper than starting over from RTL synthesis, but it's still not good.

If we're fixing a critical bug anyway, why not fix all the other bugs too? Again, this takes time -- time to figure out and implement a fix, time to rerun the design verification tests. That time means it will take longer to get the next product to market. And in the meantime, you'll almost certainly find more bugs in your current product if you look hard enough. It's a losing battle. Fixing bugs is even harder on a product that's been out for a long time, since people have to dive into the old design to figure out what's going on. As Null says, customers may have to requalify your product in their system. If your product is still in development, delaying the production release may cause customer schedules to slip, which makes customers very unhappy.

Normally, the bugs that get left in only happen in weird configurations, cause very minor problems, have easy workarounds, or all of the above. They're just not bad enough to be worth the trouble. And if you reuse a hardware module on the next product, your existing customers will already have the workaround in their software anyway.

Software toolchains are another factor. If a module sticks around long enough, your toolchain might change enough that redoing the old validation tests becomes a major project in itself. And you probably can't just load up the old tools, because you're not paying for the site license anymore. But as long as you don't change the module, you can keep copying and pasting it into new MCUs.

Software is also an issue on the customer side. If your bugfix breaks backwards compatibility in any way, all of your customers will have to update their code, which they may not even have the tools for anymore.

As someone who works in microcontroller development, I can tell you that we would all love to fix every bug. But trying to do so would delay development unpredictably, annoy customers, cost a ton of money, and at the end of it all, we'd still probably fail.

Answer 2

It's generally because of expense.

There's always a risk of breaking something else when you "fix" a bug. Because of that, the manufacturer typically needs to completely re-qualify and re-characterize the device just to make sure the "fix" hasn't introduced a different (and perhaps even more undesirable) bug. That means money and time (which, for the manufacturer, is also money). It also means the manufacturer has employees fixing an existing product instead of developing a new one.

On a related note, sometimes customers also require re-qualification of the fixed device in their product(s) to make sure that the bug fix doesn't break something in their system, either. That costs money and time for them, and customers may not be willing to accept those costs -- they'll still demand the "buggy" version.

In some cases, of course, the bug really is technically difficult to fix. In that case, it's even more expensive to fix it.

Answer 3

If a major purchaser of a part uses it in a design they've certified for e.g. use on board an airplane or space craft, any change to any of the components used in the design will require recertification of the design as a whole. If the design adequately works around all the bugs in the silicon, revising the silicon may require either having the customer redo all of the qualification testing for his board, maintaining a supply of both "non-fixed" and "fixed" parts, or simply continuing to manufacture the old design. Chip vendors do not publish their purchaser lists, but in some cases a single customer may represent a sufficiently large fraction of the demand for a particular chip that the company may be loath to do anything to inconvenience that customer.

That having been said, there are some silicon errata that keep appearing in succeeding generations of parts, some of which lack decent workarounds. Probably my biggest peeve is with a race condition in the transmit logic the UART in Microchip's 18Fxx parts which can cause it to transmit spurious NUL bytes if code attempts to transmit data at just the wrong time. Microchip's suggested workaround is to have code ensure that it doesn't try to load the transmit-data register between the time while the UART starts to send the stop bit for an earlier character and the time such transmission is complete, but if interrupts are ever disabled, code in a transmit-buffer-empty interrupt handler generally won't have any way of knowing how much time might have elapsed between the transmission of an earlier character (which would have caused the a transmit-buffer slot to become available) and when the interrupt handler actually received control, and thus won't have any way of knowing when it needs to refrain from transmission.

While I can understand how bugs like the Microchip UART bug could sneak in, the fix shouldn't be difficult: I expect Microchip generates a "go" signal based upon the "AND" of non-synchronized "transmission complete" and "character loaded" signals, and has trouble if the former signal changes state just after the latter (causing the TX buffer circuit to miss a chance to load the character data on a given cycle, but allowing the TX sequencer to start a new transmission on that cycle); even if Microchip doesn't want to add synchronization delays to the normal cases where the transmitter is empty and a character is loaded, or where the transmitter becomes empty after a character has been loaded, the problem could be fixed without affecting the timing in either of those cases by adding three NAND gates and two synchronizing latches. Numerous parts, however, have shipped since that problem was published, without adding any such fix.

Answer 4

It really depends on the company and the complexity of the fix. For example, see this errata for the PIC18F23K22. You can see that there were eight known bugs which affected the first ("A1") revision of the silicon.

At the time of this answer, they have one updated "A2" revision. Of the original eight bugs, three of them have been corrected in this new rev.

Another deciding factor is the manufacturing lifetime of the product. Even if a manufacturer chooses to not fix a specific problem in an existing part, they can still "solve" the issue by making sure that new products don't have the same bugs.

Answer 5

Maybe they already have produced (but not yet sold) thousands or millions of the ICs when a bug is found. They don't throw them all away just because of a bug.

I think you can compare it to book printing. Books are printed in numbers of many thousands in one run within a short time (days, weeks). But they are sold within years or decades. The books are not thrown away and reprinted as soon as a typo or other error is found. Also for books errata sheets are printed and handed to the user.

Of couse the known bugs (typos, errors) will be fixed in the next edition.