3
\$\begingroup\$

I'll soon be deploying a new embedded system. I have a separate serial port that can connect to a diagnostic program. I also have two LEDs, one I flash to indicate the OS is running, the other that the application is running. I fear that it's going to be hard to work out what happened if the embedded system experiences a total crash however. It does have a watchdog so would reboot, but I'd like to be able to work out why the crash occurs, if it does.

The only way I've done this in the past is to have a parallel system back in the lab, give it the same type of inputs, try and trigger the problem, then analyze through debug prints or outputs to digital I/Os to figure out what could be causing it. Often it is very hard to replicate the problem however.

Does anyone have any advice on good methods they have for debugging in field problems?

Thank you, Fred

\$\endgroup\$
7
  • 2
    \$\begingroup\$ You mention an "OS", what are you using? If it has a persistent file system then having a log file is what first comes to mind. If not... then maybe edit your question with some more specs of your system and I'll write up an answer. Also, are you interested in being able to debug and diagnose remotely? Or would you want to recover the unit and then do a post-mortem analysis? \$\endgroup\$
    – Jon L
    Oct 3, 2011 at 19:24
  • 2
    \$\begingroup\$ I'd second the 'log to permanent storage' option. It probably won't do any good for a total crash, but when your code detects an anomaly (deadlock, assertion failure, etc) it can log first. If you can spare the time and RAM you could write some circuilar log while running. When the log is present at startup, write it to the permanent storage. The 'place where it stopped' might give some clue. In general, when debugging a remote embedded system, ANY clue you can get is worth gold. But I guess you already experinced this. \$\endgroup\$
    – Wouter van Ooijen
    Oct 3, 2011 at 19:32
  • \$\begingroup\$ The system is running a bare bones OS for machine control. There is no filesystem, but we do have some spare RAM and do have Flash memory that can be accessed by block. Being able to diagnose remotely would be ideal. These embedded systems are connected to a PC and we have logmein we can use to access the PCs from anywhere. \$\endgroup\$
    – fred basset
    Oct 3, 2011 at 19:44
  • \$\begingroup\$ RE Wouter's comment; if you where logging to RAM then wouldn't you lose that log in the event of a crash? And then it's usually impractical to log to Flash as you have Flash wear, wear levelling etc. to contend with. \$\endgroup\$
    – fred basset
    Oct 3, 2011 at 19:46
  • \$\begingroup\$ If they are hooked up to PC's that are networked then I would write a logging application for those PC's that capture debug messages from the embedded device's serial port and log them to a file. If something goes wrong, you can log in to the PC remotely and check the log file. If you have many devices to one PC, consider using USB to Serial port cables and a USB hub in order to have multiple serial ports. \$\endgroup\$
    – Jon L
    Oct 3, 2011 at 20:06

1 Answer 1

Reset to default
3
\$\begingroup\$

Recently, I deployed a few units of a new embedded product for a friendly trial. I was trying to discover how well the IP stack performed out in the real world, so using the on-board ethernet for debug wasn't an option.

I bought several serial dataloggers from Sparkfun (many similar devices are available) and attached one to each unit over TTL serial. When I got the units back, I retrieved the logs from the SD cards.

In the case of a system crash, dumping registers out to the serial port will let you find the offending code via the map file later.

\$\endgroup\$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.