# Distinguish between hardware or firmware issue

I'm writing a firmware for CC3220SF from TI using a custom board I've designed. I'm facing a weird behavior and I'm not sure if it's hardware or firmware related.

Here I'm looking for some hints about how you would investigate further in such a situation.

Basically, my application follows this pattern:

1. wakes up from hibernation
3. acquires some sensors
4. sends them to a Cloud
5. read again the RTC if needed
6. goes back into hibernation

nothing new, I know. The problem is, sometimes, the readings from RTC are wrong. With "wrong" I mean during subsequent calls (even in the same active window) the read values are in the future (i.e. 2022) but after that come back to the correct ones.

An example from my logs:

[SYS] SlDateTime_t: 2017/11/28 23:16:5
[SYS] struct tm: 117/10/28 23:16:5
[SYS] time_t   : 1511910965
...
[SYS] SlDateTime_t: 2022/3/1 9:58:17
[SYS] struct tm: 122/2/1 9:58:17
[SYS] time_t   : 1646128697
...
[SYS] SlDateTime_t: 2017/11/28 23:16:9
[SYS] struct tm: 117/10/28 23:16:9
[SYS] time_t   : 1511910969


here the code I'm using to test it:

time_t _GetEpoch(void)
{
_i16 ret;
_u8 pConfigOpt = SL_DEVICE_GENERAL_DATE_TIME;
_u16 pConfigLen = sizeof(SlDateTime_t);

SlDateTime_t dateTime = {0};
ret = sl_DeviceGet(SL_DEVICE_GENERAL, &pConfigOpt, &pConfigLen, (unsigned char *) &dateTime);
ASSERT_ON_ERROR(ret);

struct tm t;
time_t t_of_day;

t.tm_year = dateTime.tm_year - 1900;
t.tm_mon = dateTime.tm_mon - 1;
t.tm_mday = dateTime.tm_day;
t.tm_hour = dateTime.tm_hour;
t.tm_min = dateTime.tm_min;
t.tm_sec = dateTime.tm_sec;
t.tm_isdst = -1;
t_of_day = mktime(&t);

UART_PRINT("[SYS] SlDateTime_t: %d/%d/%d %d:%d:%d\r\n", dateTime.tm_year, dateTime.tm_mon, dateTime.tm_day, dateTime.tm_hour, dateTime.tm_min, dateTime.tm_sec);
UART_PRINT("[SYS] struct tm: %d/%d/%d %d:%d:%d\r\n", t.tm_year, t.tm_mon, t.tm_mday, t.tm_hour, t.tm_min, t.tm_sec);
UART_PRINT("[SYS] time_t   : %lld\r\n", t_of_day);

return t_of_day;
}


My thoughts:

• I'm asserting against errors returned from sl_DeviceGet(), hence the function is executed correctly
• from the logs you see the error is in the SlDateTime_t structure that is returned by the sl_DeviceGet() function, hence is not related to mktime or my other part of the code
• on the other hand, if the next reading is correct, it doesn't seem an hardware problem because I would expect a corruption of the values also in the next readings. Instead they are correct.

The last test was an infinite loop where I read 100 times the RTC, then go to hybernation and again. The results confirmed that one reading among the 100 is wrong. The others are correct!

## UPDATE

After puzzling over a bit, I discovered the following:

• with the test code I got 250 wrong values in less than 60k readings
• the difference between a wrong value and the immediately before or next correct reading is always 134.217.728 seconds in the future.
• Which is a power of 2, giving a clue what's wrong. You may want to read the HW manual and the sl_DeviceGet source with a view to finding timing hazards during the RTC's internal updates or the syscall reading its registers. And possibly the errata for that specific CPU. – Brian Drummond Dec 7 '17 at 16:31
• About the errata, this is the TI's official answer: "We don't currently provide an errata sheet. " – Mark Dec 7 '17 at 21:03

Here are some rules of thumb:

1. It's usually a firmware problem.

2. Intermittent hardware failures can often be made less intermittent by changing the temperature, voltage, clock frequency, or other operating conditions. (Some firmware errors are like this, if their behavior depends on the state of the hardware.)

3. Gross hardware failures are usually pretty straightforward. If something basic is broken, and replacing the unit fixes it, it was a hardware failure.

4. Know the hardware and the code. A good knowledge of the interacting parts of the system is vital.

5. Sometimes a firmware error can look like a hardware problem (e.g. timing violations). Adding delays in your firmware or lowering the CPU frequency can sometimes help pinpoint the error.

6. Try to narrow down the exact subsystem that's failing. If it's an internal module, make sure your register reads and writes are working properly. If you're communicating over a serial bus, check the bus with an oscilloscope.

7. If your firmware produces a bad decimal value, convert it to hex to look for flipped bits and uninitialized bytes.

In your specific case, it looks like one bit in the most-significant byte is getting flipped (0x08000000). This could certainly be a hardware error, but to prove it you'll need to verify that you're talking to the module correctly. I'd suggest first checking the datasheet to make sure you've configured the RTC properly and aren't violating any constraints. If you have another CC3220, see if you can reproduce the failure on it. If not, try slowing down the clock to see if the failure is timing-related. If you can tweak the supply voltages, try running at the min, max, and nominal voltages on each supply and see if the failure rate differs. Looking at the source code for sl_DeviceGet() could also be helpful.

If you try reading the RTC as fast as possible, do you see many failures in a row (which could mean a brief period of hardware failure), or are they more spread out?

• Thanks for summarizing up all the stuff. I did most of your points. About the faster readings of RTC it definitely are more spread out. I mean, in operative conditions I have a RTC reading every one minute and in few hours I get a wrong value. But if I read the RTC continuously the wrong value appears (alone) more rarely (i.e. I got 250 wrong values on 60k) – Mark Dec 7 '17 at 19:05
• That's a tough one. If you're using a TI-supplied driver, you could try searching and/or posting on their E2E forum. – Adam Haun Dec 9 '17 at 16:37
• Yes, I've already did, with no answers. – Mark Dec 9 '17 at 16:38