Over the past couple of years we have done some minor revisions to our PCB, including

  • Different component selection (resistors / caps / ...) based on availability / price (while still taking into account their original characteristics)
  • Components layout on the PCB
  • Trace widths / lenghts (as a result of the layout)

What we've often found is that although these changes shouldn't really impact the embedded software, we still needed to tweak our software after each of those revisions.

Some hardware revisions brought instability with them that only showed up after days / weeks / months (ex: the inability to power toggle a certain component on the board)

We needed to tweak high/low sequences, serial line timeouts

The big poblem is that these issue don't show up immediately, but sometimes take weeks/months before they start popping up, making it difficult to perform corrective action (especially if one of the issues is the failure to power toggle the modem needed to perform a firmware update).

Are there any guidelines / best practices to eliminate that risk ? (some kind of stress testing procedure / things we need to take into account while doing such revisions ?

  • 1
    \$\begingroup\$ These guidelines are called testing, in some companies they have stuff like "QA departments". \$\endgroup\$ – PlasmaHH May 12 '16 at 13:26
  • 1
    \$\begingroup\$ And I guess a proper initial design helps too... Of both HW and FW. Let it do to experienced professionals, or at least have them to supervise the process. \$\endgroup\$ – Eugene Sh. May 12 '16 at 13:29
  • 2
    \$\begingroup\$ You have to perform testing and quality assurance. There is no magic recipe to produce a flawless product or not to introduce regressions over revisions. \$\endgroup\$ – Eugene Sh. May 12 '16 at 13:34
  • 1
    \$\begingroup\$ not exactly related to your question, but an issue I found was keeping concurrent software version with different hardware in the field. Having 6 hardware versions was a software nightmare. We used an AT91 that had an ADC pin that we weren't using. We hooked up the ADC to a resistor divider that changed between revisions. This $0.01 change saved us a lot of time on the software side because we could have different code sections based on reading this resistor divider via the ADC. \$\endgroup\$ – b degnan May 12 '16 at 13:47
  • 1
    \$\begingroup\$ @bdegnan Good advice, but note that HW identification is only a small part of the problem with supporting several HW platforms. You still have to test each fix on all HW versions, and some fixes will require you to implement different code for different HW. \$\endgroup\$ – Dmitry Grigoryev May 12 '16 at 13:54

The "best practice" here is called regression testing. Simply put, you want to have a batch of tests which

  • cover the core functionality of all components of your system
  • can be executed with minimum human intervention

The second requirement is important if you want to catch intermittent issues which surface only after some time. If the test is manual, you will be able to run it a couple of times. If the test is highly automated, you can run it repeatedly for a couple of days 24/7, spotting irregular events which don't happen every time (like power toggle failures).

It's also usually a good idea to include a stress test which uses the maximum CPU time / Memory space / communication bandwidth / electrical power.

Finally, if testing is a problem for your team, consider designing a system with higher tolerances. Include a power supply which is able to provide 50-100% extra power. If you have an MCU, make sure your stack is never used by more than 50% of its capacity and the CPU is idle at least 50% of the time. This won't eliminate the risk of course, but will reduce it significantly.

| improve this answer | |
  • \$\begingroup\$ Thx a lot for the comments. We did automate a number of things during testing, ex: putting in place a testing rig that manipulated the device under test in various ways (input voltage swings, reboots, forcing sleep mode / triggering wakup). But its difficult to reproduce an issue that only popups up after +1 month. And even more difficult to asses if its hardware of software related. We tried making it fault tolerant by providing power toggles on all major components (gprs / gps / ...) Perhaps we should have outsource that part and worked together with a specialized shop for that. \$\endgroup\$ – ddewaele May 13 '16 at 6:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.