STM32F7 device freeze: cannot access registers

Question

I am encountering random freezes with the STM32F7. This issue is difficult to debug as any debug session launched in Eclipse crashes when trying to halt the core, which makes it impossible to see where in the code the freeze occured.

Similarly, I am not able to access the device registers to inspect the program counter, the stack pointer and the IPSR, either via the eclipse debug session or using the ST-Link Utility software (the device resets when connecting and the state is lost).

I would appreciate any information on how to extract the device state when frozen, or whether any states can be recovered after a reset. Also if this situation sounds familiar to anyone, please let me know if you found a root cause. I am available to answer any questions and provide more detail.

additional details

(a) The hardware - I'm afraid I can't give you too much info here, it's a custom proprietary design.

(b) The history - This freeze issue has only become a problem recently. When I retested our older code however, the issue was seen, albeit less frequently (I think). I have been reverting bits of code to see if the issue still occurs. The issue is that I have no logs or debugging capabilities when the device freezes, so I cannot pinpoint where the code is failing.

The issue occurs infrequently, so it is difficult to say whether or not a change has had any effect.

(c) I haven't tried reducing the (system?) clock speed, and I will retest the device more thoroughly with different connections removed.

(d) Failure rate is roughly 1 failure per device per day, but it is difficult to quantify any improvement/regression (device is running all day).

(e) You ask about capturing data after a reset, but you report the problem is the MCU "freezing", not that it is resetting. I guess (but it isn't clear) that the problem is always that the MCU freezes. It's you who can perform a manual reset, and you are asking whether any useful info can be captured after manual reset. Is that correct?

Correct. Retrieving info during the freeze would be even better though.

Debug steps

(i) This problem is not reproducible with a test despite my best efforts.

(ii) what you are doing after that e.g. are you performing a manual reset?

Trying to read the registers, pinpoint where the code hung.

Answer 1

If the debugger is locked out, assuming power/clocking is OK, then the most likely outcome is a internal bus deadlock. As an example, if you have external XIP EEPROM and you have mis-configured the device, the peripheral could hang when you hit specific address/data patterns.

You may still be able to read RAM information after reset (or add some tracing to dump something to RAM). The core architectural registers are not reset (unless you have a high-reliability safety specific part), so if you put a B self in the reset handler (assuming the debugger 'halt after reset' doesn't work) then your old state should be preserved.

You can use ETM trace to capture realtime trace, and if you capture trace into the ETB you can just read this out after reset (again, it should be persistent) without needing a probe (but maybe the buffer pointer will be lost, so you might need to manually process the trace). ETM trace will show branches (addresses for any indirect branch) and exceptions (including lockup) - so you should get a reasonably accurate indication of what the last thing was. Speficically, the trace is just 'watching' the core, so it sees instruction retire without being affected by the bus activity. The DWT might give something useful too - depending on what actually fails.

As referenced in one of the comments, at least using MDK, you are able to attach to a 'hot' running target (without reset/halt, without code download), since the debug port is a completely asynchronous bus master embedded in the core. This gives you the ability to probe memory state on a long uptime device (but not core registers) with minimal intrusion, and this might be valuable to confirm a diagnosis. You could even script up a regular polling of memory locations if this is relevant.

Finally, since the debug port 'acts like the core', you are able to mimic quite a lot of bus deadlock failures by just using the debugger memory window.

Answer 2

I will recommend that you dont arrive at any conclusions right away. Since your design is proprietary, we dont know much about the hardware. So I can suggest you some general troubleshooting tips.

1. The issue could be a short somewhere on the board or an open connection, or maybe a loose connection. We don't know what it could be. So first thing, clean the entire board with isopropyl alcohol (aka PCB cleaner), give it a nice rub with a toothbrush and then use Kimwipes to wipe away the dirt/dust/metal pieces etc. Check if the uC works. If it doesn't, then inspect the PCB post cleaning for any broken traces, damaged PCB pads, cold solder joints etc. Preferably, at this stage, you should try reworking the entire board to eliminate the "loose connections/joints" issue. Check if the uC works after this.

2. If the uC still doesn't work, check for basic things like power supply fluctuations, noise on the signal lines, clock pulses of the uC etc. Use a DSO to monitor the clock pulses and reset line. Also check for continuity on all connectors. Wiggle the wires when the uC is running and see if the issue is fixed. Verify these things and then give it a go at the uC again.

3. If that doesn't fix the issue, then check the software. Flash the code on new uC (not a new development board!). Make your own barebones dev board. By barebones, I mean no peripherals attached, no external circuitry. Just the clock and some regulators thats it! You need to make sure that the code isn't freezing the uC. Eliminating the external peripherals will eliminate the problems cased by them (if any). So now this puts your software under test. When you make your own barebones dev board, flash a blink LED code to make sure its working. Then flash your software on the uC and check whether the uC freezes or not. If it does then be happy, it was the software causing the trouble. Make necessary changes in the s/w and make it work on the barebones dev board. Once it works perfectly on the bb devboard, switch to the PCB or the actual development board you are using.

4. If the uC still freezes, then it means you may have altered some 'not-to-be-touched' register bits of the uC via the software. Check your software and note down all the register and bits it is changing. Verify that the s/w makes use of the uC in a safe manner.

5. If the issue still persists, check whether the flash tool you are using is working properly or not. Do this by using the flash tool to debug "healthy" uC boards. If the operation is successful, then your flash tool is okay.

6. If your s/w freezes every uC, then probably its the issue with the s/w. Check if STMm32F7 has something like "Use a high voltage parallel programmer to reset the fuse bits" provision. Bricked Atmel MCUs can be revived using HVPP.

Try out these methods and let me know what happens. Be patient in the debugging process, eliminate problems one by one. Remember, coding is a skill but debugging is an art!