8
\$\begingroup\$

I'm running a Microchip dsPIC30F6012a. I have this chip on several PCBs, all running the same software, and observe the same problem on all of them. This implies a systemic problem, not a one-off production issue. The problem is also reproducible, implying I should be able to kill it if I know where to look. But I'm still having surprising difficulty debugging the application.

The board under test accepts 24V, which gets stepped down to 5V through a V7805. The chip runs on its internal oscillator, with a 16x PLL, giving an operation speed of ~29.5 MIPS. The relevant code on this board is essentially very simple: wake up, read data from EEPROM, then enter an infinite loop. Interrupt every millisecond, observe some environmental data, and write an updated value to EEPROM. There's other stuff going on, but the problem still occurs even if the unrelated code is commented out, so I can be reasonably certain it's not relevant to the problem at hand.

In general use, 95% of the time the board wakes up with the correct value in memory, and goes on about its business. The other 5% of the time, though, it wakes up with an incorrect value. Specifically, it wakes up with a bit-flipped version of the data it's supposed to have. It's a four-byte unsigned long that I'm watching, and either the upper or lower word of the long can get flipped. For example, 10 becomes 2^16-10, which later becomes 2^32-10. I can reproduce the glitch by manually cycling power several dozen times, but that's not very consistent, and my switch finger gets worn out.

In order to reproduce the problem in a controlled fashion, I built a second board which drives the 24V supply to the board under test. (Another dsPIC driving a darlington optocoupler.) The tester board turns the 24V off for 1.5 seconds (long enough for the 5V rail to drop to essentially 0 and stay there for one second), then turns the 24V on for some configurable length of time. With an on-time of approximately 520 mS, I can reproduce this EEPROM glitch within five power cycles, every time.

The 5V rail is behaving reasonably. It settles at 5V within 1 mS of turn-on, with perhaps .4V of overshoot, assuming I can trust my scope. At turn-off it decays to 0V exponentially, reaching 1V within 50 mS. I have no build warnings that seem relevant, just unused variables and missing newlines at the end of files.

I've tried several things:

  • Enabling/disabling the MCLR
  • Enabling/disabling the WDT
  • Enabling/disabling code protection
  • Enabling/disabling/changing brownout detect voltage
  • Enabling/disabling/changing the power-on timer
  • Different PLL settings on the main internal oscillator
  • Connecting/disconnecting my PICkit 3 programmer
  • Adding 470 uF of capacitance to the 5V rail
  • Adding/removing .1 uF across the 4.7k pullup on my MCLR pin
  • Disabling all interrupts in the code and leaving nothing but EEPROM updates in the main loop
  • Adding a 1.5 second delay to my startup routine before I start reading EEPROM

I've also written separate test code which does nothing but continually write values to EEPROM and then read them back, making sure that the value has not changed. Tens of thousands of iterations gave no errors. All I can conclude is that something goes wrong with EEPROM read or write, specifically at powerup/powerdown.

I've been using the same EEPROM libraries since 2007. I've seen occasional glitches, but nothing reproducible. The relevant code can be found here:
http://srange.net/code/eeprom.c
http://srange.net/code/readEEByte.s
http://srange.net/code/eraseEEWord.s
http://srange.net/code/writeEEWord.s

I've seen EEPROM errors before in other applications, but always as one-off glitches, nothing this reproducible or consistent.

Does anyone have any idea what's going on? I'm running out of things to try.

\$\endgroup\$
3
\$\begingroup\$

Two things come to my mind:

First, according to the data sheet, a erase-write-cycle takes at least 0.8ms, and up to 2.6ms. You say that you have an interrupt every 1ms, which may lead to a write operation. I have seen in the code that you disable interrupts for parts of the erase and for parts of the write function. But you still might get funny interleaving of the function calls. Maybe it helps when you disable interrupts for the whole sequence of erase and write?

Second - you might want to write while to power goes down, and the EEPROM write happens exactly in the moment when the supply voltage goes below the operating voltage. You can try to monitor the supply voltage, and refuse a write when it is below, lets say, 4.5V. This assumes that it stays long enough above 2.7V as the minimal operating voltage, and brown-out-detection is set to trigger only below that point.

\$\endgroup\$
  • \$\begingroup\$ You're close! The cycle is erase->write, so if power-down comes between erase and write, you lose your data. My first solution was to divide the EEPROM into multiple redundant copies, which are automatically checked for inconsistencies. But since that ate 3/4 of my EEPROM, I'm replacing that with a simple write buffer. The buffer will be a special EEPROM block that contains the data to be written, the address to write to, and a flag indicating that the write is not yet complete. This should address the issue while taking much less space. \$\endgroup\$ – Stephen Collings Feb 4 '13 at 14:31
  • \$\begingroup\$ I can now confirm that my buffer-based approach works, and does not suffer data loss due to asynchronous power-down between erase and write. \$\endgroup\$ – Stephen Collings Feb 11 '13 at 21:03
4
\$\begingroup\$

You have looked at a lot of possible hardware issues. That's fine, but this is most likely a firmware bug.

Unfortunately, your source code is poorly documented and is formatted to be difficult to follow visually. Your first file contains declaration of external routines at the top:

void readEEByte(unsigned int, unsigned int, void*);
void eraseEEWord(unsigned int, unsigned int);
void writeEEWord(unsigned int, unsigned int, void*);

Not only is it a bad idea to put declarations like this private in client modules, there is not a single comment in sight! We can only guess what you intend these routines to do from their name, and the call arguments are completely undocumented. Further in that file you have various lines starting with "//" and a whole line of equal signs. These add visual clutter making it too much trouble to try to follow the code.

You may say that none of this matters for the operation of the code. However, bad programming practises like this do matter, a lot. They cause code to be written poorly and make it hard to spot bugs or even what the code is supposed to do. All this results in lurking hard to find problems, as you are discovering. You even said yourself you have occasionally seen glitches from this code since 2007. That should have been a strong clue of a bug, possibly even of bad overall design.

Fix the mess, properly document all interfaces, and put common declarations into include files that you write once, then reference when needed. Also your statement of I have no build warnings that seem relevant is a huge red flag. Again, fix the mess. When debugging, always go after the easily reproducable and fixable problems first. Sometimes those are actually the cause of the tough problems, or sometimes in fixing them you discover the cause of other problems. The compiler is warning you of sloppiness on a silver platter. What more do you want? You shouldn't have unused variables because they cause confusion to anyone trying to make sense of your code, and there is no excuse at all for missing new lines. Again, fix them obvious mess, especially before asking anyone else to look at your code.

Neatness and attention to detail matter. A lot.

 

\$\endgroup\$
  • \$\begingroup\$ You're quite right about the code. Just so we're clear, I started using this code five years ago, but someone else wrote it. I don't write stuff that looks like this. I should still fix it, and it's not good that I haven't. Just so I don't look like QUITE so big a goofball. :-) \$\endgroup\$ – Stephen Collings Aug 6 '12 at 14:40
  • 2
    \$\begingroup\$ Accessing the internal EEPROM is easy. For such simple stuff, it's easier to write your own code than try to figure out how someone else's bugware works, then patch it up untill it seems to work. Read the data sheet and write the code. You'd be done in a hour. \$\endgroup\$ – Olin Lathrop Aug 6 '12 at 14:50
  • \$\begingroup\$ I agree with Olin here in that it's most likely firmware. The part errata mentions nothing is fishy with the EEPROM. \$\endgroup\$ – Adam Lawrence Aug 6 '12 at 15:19
  • 2
    \$\begingroup\$ @Madmad - Errata sheets may not say there's something fishy with the part, but they'll never say there isn't anything fishy with it :-) \$\endgroup\$ – stevenvh Aug 6 '12 at 18:06
  • 1
    \$\begingroup\$ @stevenvh My dealings with Microchip and their FAEs have been mostly positive. The errata for the parts I've been using have been accurate, and they've jumped in to help us with issues, often finding workarounds for us. \$\endgroup\$ – Adam Lawrence Aug 6 '12 at 22:15
0
\$\begingroup\$

I had an identical behavior with 4 pcs of dsPIC30F6014A (of about 10 used over the last few months..), the only way to avoid the sporadic data corruption during power off is to set to zero the MCLR just before shutdown.

Obviously this is not feasible in practice, so I opted for the replacement of the "bad" dsPIC, if anyone has another solution instead ...

\$\endgroup\$
  • 3
    \$\begingroup\$ Why wouldn't that be feasible? Detecting power outages is done a lot, even for saving data to EEPROM at the last ms. \$\endgroup\$ – stevenvh Sep 12 '12 at 16:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.