0
\$\begingroup\$

I have a project running off a Bluepill board (STM32F103C8T6) that ceases to function only after it runs for ~6 hours. Up to that point it behaves totally normally, and UART logs don't indicate anything out of the ordinary until it just stops. This has made it very painful to debug since it's so difficult to gather information and attempt iterative solutions.

The only lead I have is that when I find the device in the hang state, it cannot be reset properly. In the first few (correct) hours of operation, rebooting the device plays a speaker and blinks an LED. After the hang, resetting results in a solid LED - no sound, and no operation past that point.

Only when I power cycle the device does it return to normal operation, and then everything works exactly as I want it to for another ~6h before it hangs again.

Conventional wisdom would suggest this is a memory leak or a timer overflow, but I don't see how either of those could persist past a system reset! The Bluepill is attached to I2C and SPI devices, but even when those are removed from/returned to the circuit it does not reset properly, and there's nothing else in the circuit that can hold state. I've also heard that an unconnected VBAT could cause this but it's connected to VDD, and reading the datasheet shows no unexpected NRST behaviors.

I'm reluctant to post all 250 lines of code for this project because I haven't been able to narrow down the cause of this at all, so I'm trying to stay general with this question. What kinds of code bugs are capable of surviving an MCU pin-based reset, but not a power cycle?

EDIT: Here's the schematic. VBAT was manually connected to the 3.3V pins later. bluepill breakout

EDIT: This is terrible form for stack overflow but here is a pastebin of the code. I hope to condense this to only the problematic code ASAP but due to the massive repro time I've had issues narrowing it down. https://pastebin.com/nKDyzRKa

\$\endgroup\$
  • \$\begingroup\$ IMHO this could have a hardware cause. Further analysis would require you to edit your question and add (a) a complete schematic (or at least a clear and complete block diagram) of the minimal configuration which still shows the problem, and (b) some photos of that same minimal "failing" configuration. \$\endgroup\$ – SamGibson Apr 25 at 16:35
  • \$\begingroup\$ I did not include a schematic as I noted that the problem can be isolated to the Bluepill, which is a generic board that simply hosts pinouts and oscillators. But I will post my schematic to clarify. \$\endgroup\$ – Hierophect Apr 25 at 16:42
  • 1
    \$\begingroup\$ Hardware but not thermal. If it was thermal you'd have to let it cool, at least for a bit. Can you measure the supply current? You may be wedging a power supply. Or you may have some external peripheral that's getting wedged, and isn't getting properly reset when the processor comes back up. \$\endgroup\$ – TimWescott Apr 25 at 16:43
  • \$\begingroup\$ @Hierophect "as I noted that the problem can be isolated to the Bluepill" (a) It wasn't clear to me that you had removed absolutely all external connections. (b) Sometimes we see things in photos & diagrams which lead to answers, even without originally knowing what the photos could show when asking for them. (c) It's possible that this "BluePill" is faulty - either from new, or due to earlier attached devices. Have you tried another one? || But anyway, it's your question so it's up to you how much information you supply (which can affect how much help you receive). Over to you :-) \$\endgroup\$ – SamGibson Apr 25 at 16:48
  • \$\begingroup\$ I have added the schematic for clarity. @TimWescott, I expected that might be the case (particularly with the NRF24) but I've found that the hang state behavior still does not match regular behavior even with them removed in both cases. \$\endgroup\$ – Hierophect Apr 25 at 16:54
2
\$\begingroup\$

Maybe something is being clobbered that you are not initializing in your code and that is not automatically initialized by the reset. When the power is removed it may take on a random value that happens to be okay.

250 lines of code is not very much, but maybe you can further reduce it to try to isolate the problem. If you have a debugger attached to the system when it hangs, the cause could be related to the external system.

\$\endgroup\$
  • \$\begingroup\$ Unfortunately, a debugger is impossible in this case because the device spends 99% of its time sleeping, and so the conditions under which it hangs cannot be replicated with a debugger attached. \$\endgroup\$ – Hierophect Apr 25 at 16:44
  • 1
    \$\begingroup\$ @Hierophect - FYI, one example of the approach mentioned by Spehro for reducing the amount of code in your test case, is explained here: How to create a Minimal, Complete, and Verifiable example. \$\endgroup\$ – SamGibson Apr 25 at 16:50
  • \$\begingroup\$ Be sure to read the errata for the chip, especially with regard to sleep mode, and make sure you're allowing enough time for things like the clock to wind up. The startup from sleep may take a crystal longer than when you whack the power onto the Pierce oscillator circuit. \$\endgroup\$ – Spehro Pefhany Apr 25 at 16:56
  • 1
    \$\begingroup\$ @Hierophect - "Hence why I have tried to phrase the question conceptually." Beware that such conceptual questions, when you have a specific problem now, don't work very well on this Q&A site - it isn't a forum. For example, at a high level, I can tell you that I have seen both hardware and software causes of similar behaviour - but that doesn't help you on its own. And then to get into the details, requires to-and-fro in comments for measurements, scope traces etc. and suggest changes, and similar for the code. This site doesn't work well for long comment threads. Sincere good luck anyway! \$\endgroup\$ – SamGibson Apr 25 at 17:12
  • 1
    \$\begingroup\$ "Unfortunately, a debugger is impossible in this case because the device spends 99% of its time sleeping, and so the conditions under which it hangs cannot be replicated with a debugger attached." this is a good reason to put serial debug output in your program. Then you can connect to the UART, hit the reset, and see where it gets stuck. Or you can leave a few of them running 24/7 connected to a cheap computer or raspberry pi collecting the logs. Added bonus with the pi that you can use the GPIO as an SWD programmer and thus do remote or automated tests on hardware! \$\endgroup\$ – Chris Stratton Apr 25 at 20:10
0
\$\begingroup\$

The microcontroller datasheet p. 59 mentions a few things that EMC hardened software might worry about:

  • Corrupted program counter
  • Unexpected reset
  • Critical Data corruption (control registers...)

It also says you can provoke some of these failure mechanism by manually pulling RSTN or Oscillator pins down for ~1 second.

The datasheet also mentions some failure monitoring mechanisms (supply, clock) that could trigger an interrupt. Have you tried reading the interrupts after the failures or is the device non-responsive on all interfaces?

\$\endgroup\$
0
\$\begingroup\$

Many kind of issues will persist past reset. In your case you said there are I2C peripherals so the answer is easy. For example what happens when the reset button is pushed while MCU is reading or writing data to/from slave chip? The slave chip does not know that the CPU just went away and if the slave was pulling SDA pin low (data or ack bit), it will continue to do so forever. If the MCU just initializes the I2C peripheral as usual and tries to start communication with the slave, it can't generate a start condition as it regards the bus being not idle as SDA is not high. So instant jam situation after reset, until power cycling or resetting the slave. Therefore the results of I2C operations should be checked after a timeout expires. Then you can blink leds differently to signal an error. Or perform a sequence to bring the slave out of that situation manually.

\$\endgroup\$
  • \$\begingroup\$ Well indeed, that would depend on how they were removed and reinserted. If I2C devices have power, and only SDA and SCL are disconnected and reconnected, they would still keep state and keep the bus jammed. If completely removed, they would not keep state. But when removed, we can't also know how the program handles a situation where I2C devices are removed, but firmware still tries to talk to them. Also, we have no idea how the bugs in STM32 have been worked around, as it has a serious I2C analog filter problem that will also persist over a reset. \$\endgroup\$ – Justme Apr 25 at 20:54
  • \$\begingroup\$ In keeping with this being a terrible form SO post I've added the code in a temporary pastebin that I'll hopefully replace as soon as I've run another test. As an explanation of the I2C behavior: after a hang, resets only cause a solid LED. After removing i2c devices and pressing reset, it still only shows a solid LED. After reinserting i2c devices and cycling power, normal blinking and speaker noise. After removing i2c devices right after that and resetting once more, slower blinking as setup operations time out, and speaker noise. \$\endgroup\$ – Hierophect Apr 25 at 21:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.