3
\$\begingroup\$

I'm using at91sam3s8b(cortex-M3) for my project.The hardware board connects to PC using USB port. The firmware contains some cryptographic algorithms (it includes shifts, Xors, memory substitution operations). The algorithm output is correct when the board is powered on but it starts to fail as time passes.

The master clock is configured at maximum (64 MHz).

my observations are:

  1. if I lower the MCK frequency the failure rate decreases. (it's not related to the frequency decrease rate. The failure completely fades if I use a frequency of 55 MHz)

  2. If I raise the board temperature the failure happens sooner and if I lower the temperature enough the failure fades again!

  3. Some micro-controllers fail (the most of them). but some others do not. (even at an artificial temperature of 85 degrees of centigrade)

The malfunctioning boards and hardware are sent to Atmel for test. they conducted a few electrical tests and answered there is no problem relating to our board or their micro-controllers.

Any idea how or where I can track this Issue? Any technical suggestion?

Edit:

More information:

  • The project is being built using keil 4.7
  • The cryptographic routines are implemented in assembly and c and are linked to the main project as static libraries (there is 2 separate libraries. assembly is built using code sourcery).
  • changing the order of libraries in the project displaces the fault to another algorithm. changing the name, place or declarations of some functions removes or displaces the fault.
  • The hardware board is simple. a microcontroller, a USB port and a SPI flash memory(winbond serial nand flash) + that little LED. The USB port and SPI flash are not used in the simplified test code which is still faulty.
\$\endgroup\$
20
  • 1
    \$\begingroup\$ At process corner (maximum temperature, minimum instruction cycle time) some internal step within the algorithm is not available within allowed time. If issue is not caused by hardware it could be caused by a race condition (design flaw in algorithm). As a diagnostic, can you try inserting delays at strategic points, such as when writing and them reading results from memory. This emulates the same effect as slowing the master clock, but if proven effective, you can fine-tune where the sync delays are required and omit them where not required. \$\endgroup\$
    – MarkU
    Commented Mar 26, 2016 at 9:16
  • 2
    \$\begingroup\$ Are you sure the wait states for the internal flash, and for any external memory, are set correctly for your operating frequency? What's the symptom of the failure? \$\endgroup\$ Commented Mar 26, 2016 at 10:10
  • 1
    \$\begingroup\$ @MarkU Since the algorithm executes in a single thread and interacts with no peripheral, the only possible race condition here is the race between instruction fetch and instruction execution time which is considered in Flash Wait State parameter. As datasheet implies using a FWS = 3 allows the microcontroller to be used at 64MHz core clock safely and this is the value I've configured. I even tried using a higher wait state. but still no luck. And as for inserting delays, keep in mind that the processor only interacts with its flash which should be safe considering what I said above. \$\endgroup\$
    – Taheri
    Commented Mar 26, 2016 at 11:28
  • 3
    \$\begingroup\$ Can you reproduce the failure (with the LED testcase) on an Atmel supplied dev board? If you can, then you have something you can throw over the wall for them to look at. If you can't, then it's time to identify what's different between the boards. \$\endgroup\$
    – user16324
    Commented Mar 26, 2016 at 11:59
  • 2
    \$\begingroup\$ @Taheri : yes but clearly they missed something! If it fails on their own board they must escalate as a matter of urgency. And if it doesn't, you have a way forward. Personally I'd be looking at the TI Hercules as an alternative. (Mention that to the Atmel rep and watch his face). \$\endgroup\$
    – user16324
    Commented Mar 26, 2016 at 12:36

2 Answers 2

4
\$\begingroup\$

@Taheri - This is not an answer, but it would need 6+(?) comments due to space limits and unfortunately comments can't have blank lines for separation of the various points. So please just treat this as a comment giving suggestions, with better formatting and more detail :-)


From experience, much more detail (some likely under NDA) would be needed by us remote readers to efficiently troubleshoot this problem on your boards, making it difficult to achieve this via remote help. The suggestion from @BrianDrummond to eliminate your hardware by using an Atmel-approved board, is a good one (+1). I have seen similar problems due to system design mistakes, nothing to do with the MCU, which Brian's suggestion should help to eliminate.

There are some standard troubleshooting approaches which either haven't been mentioned, or haven't been concluded - here is a brief, non-exhaustive, list:

(a) Simplify the code further; it seems you've done more of this than mentioned in the original question, but yet more can be done to identify exactly where in your algorithm, the incorrect result starts. Keep reducing the amount of code and inserting fixed data values, until you cannot remove any more code without the problem "disappearing". Then look at the remaining "minimal" code.

What is unusual about that specific "failing" code, which could explain why you are seeing incorrect behaviour, but most other users of the same chip are not (otherwise, if this was easy to trigger, we would likely be flooded with people reporting it) e.g. are you using an unusual peripheral, like the CRC calculation unit, which many other people won't use in their designs? What else are you doing differently, which could explain why you are triggering this issue, yet the other thousands of similar MCUs aren't failing in the same way?

(b) Don't accept that the data at some point during your algorithm is just wrong (from code calculation? or read from RAM? or something else? it's difficult to give more precise suggestions without seeing your code...). Instead identify exactly how it is wrong, by comparing the actual and the expected values at each point (bit shift? bit flip? faulty addition result? etc), and look for consistency (or lack of consistency) on the same board and across boards. Then focus on finding similarities / differences between the groups of similarly affected and similarly unaffected boards. This specific data can help, among other potential uses, to see whether your code and observed behaviour matches with any Errata from Atmel.

(c) What is the history of this project? When did you first notice this behaviour? If previous prototypes were not affected, what is different with those, compared to the "failing" boards you are asking for help with?

(d) Consider other drastic changes e.g. running the code from RAM instead of from Flash (assuming you have a JTAG port to be able to do this more easily). If the same incorrect runtime behaviour is still observed when running from RAM as when running from Flash, then this eliminates a hypothesis of timing problems when reading from Flash (doesn't it?) and that will be one step closer to finding the root cause.

Edit: (e) Have you tried transplanting an MCU from a "failing" board onto a "non-failing" board, to see if the problem moves with the MCU or stays with the PCB and the other components on it?

Hope that helps.

\$\endgroup\$
12
  • \$\begingroup\$ Thanks sam for the detailed description. I've read it thoroughly. many of them are done already. But I will surely try the rest. next comment comes with some answers. and more information comes in the Edit section of the question. \$\endgroup\$
    – Taheri
    Commented Mar 27, 2016 at 4:09
  • \$\begingroup\$ (a) is done. until any change will remove the error. but that was not a specific change. Almost any change would remove or maybe displace the failure point. (b) is done. but not completely the same way you explained. the failing value had a lot of zeroes as I remember. I agree, it worth working more. But what is obvious to me is the algorithm code is not faulty. since the fault can be displaced to other algorithm or logical codes. \$\endgroup\$
    – Taheri
    Commented Mar 27, 2016 at 4:21
  • \$\begingroup\$ (c) The big remarkable history of the project is migration from SAM7 series to SAM3 series. this is where the failure appeared which we found out later! There exist a lot of more feature implementations that can't be treated 'not related' blindly. But it is clearly obvious that the SAM3 port of the project is faulty. \$\endgroup\$
    – Taheri
    Commented Mar 27, 2016 at 4:25
  • \$\begingroup\$ (d) running the code from RAM is interesting. How does it concern having a jtag port? can you explain? I think it is sufficient to just mark the code modules to load in ram in the Keil IDE. \$\endgroup\$
    – Taheri
    Commented Mar 27, 2016 at 4:31
  • \$\begingroup\$ (e) is done. the problem moves with the microcontroller \$\endgroup\$
    – Taheri
    Commented Mar 27, 2016 at 4:31
2
\$\begingroup\$

Just some thoughts (can't comment, insufficient rep, sorry)

Have you looked at the regulator temp? If it hits its overheat limit it can shut down just enough to cause glitches. This could be worse for weaker microcontrollers.

5v to 1.8v = 3.2V and ??? amps = ??? watts. This will be dissipated in the regulator as heat. Also it will increase as frequency is increased, since current consumption also increases with frequency.

Lastly, I didn't see mention of how you're decoupling the supply to the microcontroller, if at all.

Low ESR ceramic cap right next to the microcontroller, right?

Edit: Let me clarify this last statement. There should probably be no less than three (3) decoupling capacitors in this.

  1. On the regulator input/ground.
  2. On the regulator output/ground.
  3. On the MCU Vdd/Vss.

Number 3 is the one that is most often forgotten. All of these should be low ESR, and low inductance capacitors; self resonant frequencies much greater than 10Mhz. And every single one should be as close to their respective part as engineering will allow.

Don't forget that the MCU also probably has it's own regulator built in, and that will need decoupling. Some MCUs even break their internal regulator out (Vusb pin?) and you will need to decouple this explicitly as well.

\$\endgroup\$
7
  • \$\begingroup\$ Thanks for the information charlie. I will check on these and answer soon \$\endgroup\$
    – Taheri
    Commented Mar 27, 2016 at 6:38
  • \$\begingroup\$ the requlator converts 5v to 3.3. I was wrong back there when I said it converts to 1.8. The regulator is a TI LP2980. As the datasheet imlpies it has a -40to 125 operational voltage which in my case is surely met. Although as the datasheet states. we use a 2.2 microF tantalum capacitor which is the most recommended one. (right next to microcontroller) \$\endgroup\$
    – Taheri
    Commented Mar 27, 2016 at 10:52
  • \$\begingroup\$ @Taheri - "right next to microcontroller" Actually the output cap of an LDO voltage regulator needs to be right next to the regulator - otherwise the regulator output loop includes the inductance and resistance of the PCB up to that cap. This can risk lack of stability under transient load conditions, amongst other problems. Some LDO datasheets make the point clearer, but see page 20 of the datasheet: ti.com/lit/ds/symlink/lp2980-n.pdf "For best overall performance, place all circuit components [...] as near as practical to the respective LDO pin connections." [...] \$\endgroup\$
    – SamGibson
    Commented Mar 27, 2016 at 14:27
  • \$\begingroup\$ [...] Place ground return connections to the input and output capacitors, and to the LDO ground pin as close to each other as possible, connected by a wide, component-side, copper surface." [my bold] You then need additional decoupling next to the MCU, of course. So perhaps the point made by @Charlie is still worth considering? \$\endgroup\$
    – SamGibson
    Commented Mar 27, 2016 at 14:28
  • 1
    \$\begingroup\$ Woo, I can comment now! @SamGibson - Just to acknowledge your comments; you are correct on the decoupling cap issue. I have edited my answer to reflect your input. You are also correct on the out-of-spec condition. As you said, all bets are off if anything goes out of spec. I feel this needs to be eliminated first before we can move forward. Taheri - I agree with BrianDrummond and SamGibson, you really need to test in a factory board if you can at all help it. These boards will be thoroughly tested by a large community, which will eliminate that as a variable and give you valuable incite. \$\endgroup\$
    – Charlie
    Commented Mar 27, 2016 at 21:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.