I have this question suppose we tested the software at bench level and in production it works for some days, but If you find issues after let us say some months. Is it possible for bugs to come after many months? How to identify those test cases? Generally how to plan debugging for such cases? Any software tools which can help me? Any other suggestions? I am speaking about embedded systems for microcontrollers.
\$\begingroup\$
\$\endgroup\$
6
-
2\$\begingroup\$ What kind of issues do you see? Can you tell if they are hardware or software related? What kind of tests do you run? \$\endgroup\$– 0x6d64Dec 29, 2015 at 16:17
-
1\$\begingroup\$ I believe Boeing recently discovered an issue with 777 FADECs that was only possible if a jet was running for many months continuously -- their long term test lab was the only reason they found it. \$\endgroup\$– Krunal DesaiDec 29, 2015 at 18:36
-
\$\begingroup\$ I believe this is a common problem. We had to meet requirements for 100s of thousands of hours MTBF for our devices, but without 100s of thousands of hours available to do the testing. Accelerated MTBF testing relies on putting the Device-Under-Test into an unrealistically high-stress test procedure to simulate long term usage and wear in a short space of time. I am assured it can also be carried out for devices running embedded software. Unfortunately, I do not know how it was done. I only know that we had special build variants for MTBF testing, and someone else wrote that test code. Sorry. \$\endgroup\$– Mark ChDec 29, 2015 at 22:59
-
1\$\begingroup\$ One of the older versions of Windows (95?) had a counter overflow bug that only triggered after 48 days uptime.It was years until it was discovered. \$\endgroup\$– pjc50Dec 30, 2015 at 0:09
-
1\$\begingroup\$ @pjc50 If I remember, older versions of Windows seldom ran for 48 hours. At least it seemed that way. \$\endgroup\$– tcrosleyDec 30, 2015 at 7:56
|
Show 1 more comment
4 Answers
\$\begingroup\$
\$\endgroup\$
These bugs can be in the following cases:
1) The testing did not cover the full range of use cases, and the system encountered some untested input or sequence of inputs, engaging an untested logical path
2) The system has some time-dependent feature (something that is happening on a specific date or time, or after some specific long time.)
3) The system has some counters, which are stored in some wide variables, and then they overflow at some point causing some undefined behavior.
4) Hardware failure.
It is not easy to debug such a bugs after they happen, as it is hard to reproduce them. You can:
1) Log essential diagnostic data in order to examine it if something happens
2) Make the FW run in short cycles of time, resetting it every X days/hours/weeks.
answered Dec 29, 2015 at 16:16
\$\begingroup\$
\$\endgroup\$
1
I've seen it done with logging data stored or sent to servers over time. Keepalive signals, process monitors that restart the routine if it freezes.
Typical cases can be: - Long time span memory leaks (I've seen as slow as 2kb/h that would eventually fill the RAM) - Storage space fills up with logs - Issues with communication to other equipment, like servers. What happens if they are down ?.
If you have 100% code coverage, and tested with 100% of possible input combinations and made sure your setup is not sensitive to SEU's it should be pretty robust. But shit will still happen. The most important part IMO is the 100% code coverage.
-
4\$\begingroup\$ In many cases having all the possible input combinations is simply not possible; thats why all the test methods like equivalency class partitioning or boundary value testing exist. I would agree that measuring coverage is a good thing, but 100% code coverage using bad test cases will not help. \$\endgroup\$– 0x6d64Dec 29, 2015 at 16:24
\$\begingroup\$
\$\endgroup\$
not deterministic. there are cases already mentioned, something date specific (leap year, Y2K, etc), using an 8 bit variable or math that demotes to 8 bits for day of the year (clipping at 255 days somewhere in october, being broken until jan 1st).
some problems are analog related, like single bit errors in ram, that the expected failure rate is one failure in N units of time, which might be years, so you may have to have that long of a test plus some, or have N test fixtures (millions?) and have it fail in one unit of time, or N/2 fixtures fail in 2 units of time and so on.
some problems require multiple related or unrelated things to happen at the same time or in a particular order and that may take time for randomizers or whatever is feeding the test fixture to hit that pattern. and it may take that much time to repeat it.
the biggest mistake made is to put too much fear based code in or do too much fear based testing. the longer a company or individual engineer is around the more painful bugs they have worked through and fear seeing them again so add code/tests for that going forward. Every line of code you add to the logic or the software adds risk. So if you add a handler, or checkpoint, or test in the code or in the tester, now you need another test to test that test, and another test to test the test which is testing the test. and another and another. this by definition cannot end. I have seen this fear take a products yield down to around 5% (yes 95% failure mostly due to the tester simply not working because it was overburdened by this fear and unable to function properly. Even with a government project infinite time and infinite money you cannot overcome this. there are simply things you cannot test, and you cannot get around affecting the unit by adding a test.
check the obvious stuff, the individual components are tested separately, resistors, chips, boards, so when they come together as your product, first focus on the connections, the solder points, the connectors plugged in etc, dont initially need to worry or care about the inside logic of a chip, you want to exercise the soldered down pins (a full blown memory test exercises something that was already tested, should you waste time on that yes or no? a much shorter test will cover all the address bits, control signals and data lines). Then there may be other categories, there have always been problems with bgas, but we see in the news big companies having problems with bgas a year or two into the products life, do you, can you even, do something about testing that? maybe not. maybe it is just cheaper to replace the broken units in the field than try to instrument something to test them not knowing for a year or two if your test really did anything.
the big bugs or the tough bugs often require their own special tooling software and/or hardware to find. Some of it may be re-usable but some of it is specific to that bug, once that bug is found, much of that work now goes in the trash or some of it goes into the test fixture depending on what the bug was. The next big bug you start over. There will be big bugs from time to time, you cannot predict all of them and should not try to. As a company with some set of employees, facilities, etc you may have systemic problems that keep occuring, some you can fix with training, etc some you should actually check for for some period of time until they go away (ideally due to training, dont use an ic as a fulcrum for prying off a connector)(a design engineer/team that doesnt handle multiple clocks properly or latching or rollovers in counters)(a software engineer/team that habitually forgets to check their math, clipping or allowing divide by zeros in, etc)
Yes there are categories of problems that only time will find (sometimes but not always increased by running more tests in parallel). Some can be found with simple/available tools, some you have to craft a new tool for that bug and after found that tool has no more value. Leave some budget for the latter, having some issues that you simply cannot predict, and have to craft a tool/fixture/software to find. Take a floating point unit, 32 bit only, a = b (operation) c. Multiply out the combinations for 32 bits for each variable, then multiply that by the number of operations, figure out how much code it takes to run each operation, even assuming somehow you have all the right answers computed and tested those as right answers, how long to run that test at the speed of the processor? Now try two operations back to back with the result of the first feeding the input of the second. How long does that take. now multiply that by the number of registers in the core for each of the variables to feed that alu/fpu. pretty quickly you exceed the known age of the universe. you dont have billions of years to try every combination, so dont bother. try some and budget for something to break in the future.
\$\begingroup\$
\$\endgroup\$
One topic that has not really been addressed is programs running wild or programs locking up.
In the first case, the program may end up executing a bad instruction (usually because it jumps off into a data section instead of code). This can happen, for example in C when a function is called through a pointer and the pointer has a bad address (including 0). If you are lucky, the processor will have a exception that catches this behavior. Usually about all you can do is restart the program from the beginning. It can also happen if a program incorrectly jumps into the middle of an a multi-byte instruction.
In the other case, a program locking up, this can be caught by a watch-dog timer. If you have a WDT enabled, then it needs to be "tickled" at a rate quickly enough to avoid causing it to time out. But don't put this code into a periodic timer interrupt, for example, as that defeats the purpose. In a non-preemptive multitasking system, this usually means putting code to reset the WDT into every task, to catch tasks that don't give up control when they should.
A common cause of watch-dog timeouts, are systems where the program is waiting in a busy loop for a bit in a peripheral register to become set or reset by the hardware, and it never happens. This of course is bad programming practice, but it happens nevertheless.
Like the bad address exception, about all you can do after a watch-dog timeout is to reset the processor and start over.
For both of these cases, you might want to log their occurrence in some way (i.e. to an SD card or other non-volatile storage) if you have a way of interrogating the counts later on. This is particularly useful if you also have a real-time clock and can log the date and time too.
In one system I designed, after doing a firmware upgrade over the air, the system looked to see if it got any watch-dog timeouts for some period of time after the upgrade, and if so, reverted to the previous version of firmware.
