I will come to a question further down, but first a little background
We are struggling with reproducing a nasty bug that we have been getting reports for.
The symptoms clearly show that the RTC (a DS1305) is skipping from November 30 to April 1, the same year (e.g. backwards).
We have received enough reports as to not being able to write it off as a hardware fault or solar flare or other unlikely one-time-error. However all attempts at reproducing this behavior in-house have failed. Even with the exact same hardware and settings as was used by our customer when the error did occur.
Since it doesn't always happen, nor for all devices, it doesn't feel like a software bug. At least not acting on it's own.
Question
Any ideas for how to go about reproducing this kind of behavior, fault-finding methods, what to look for, etc.
Any one else have any experience with this kind of error?
We are aware of one other with a very similar symptom, however unclear if this is related at all.
I know there is a lot of details missing. I can't disclose any source, and simply stating everything I know will be a little to much to type; I can fill you in if you post concrete questions.
Update
Finally! We have been able to reproduce this erratic behaviour in the lab!
Pressed for time as we are, all our attempts at reproducing was started one or a few days prior to 30/11 to see how it went, and all passed over to 1/12 just fine. It was after that we noticed that all customer devices were started during october.
We can't really work with waiting over a month for reproducing, so we came up with a work-around that to my surprise actually seems to work.
By speeding up the clock!
We have replaced the standard 32.768kHz osc with a 1Mhz signal, and can now reproduce in about a day.
I'll keep you posted as to what we will discover about this.
Thank you all for excellent brainstorming. I appreciate it a lot.
Now, I'm off trying to further trim the reproduction time, and dig out more facts about it.
Solved
I have posted the root cause of this as the accepted answer.
Summary: month value used was not a valid BCD value.