Electrical Engineering Stack Exchange is a question and answer site for electronics and electrical engineering professionals, students, and enthusiasts. It only takes a minute to sign up.
Sign up to join this community
For a project, I am looking to deliberately provoke data rot on a storage unit (e.g., a hard drive or a flash drive). I understand that most modern OSes and filesystems as well as hardware have countermeasures in place to prevent data corruption. How could I set up a simple system such that one could see visible corruption on, e.g., image files within a week?
I am trying to provoke actual physical data rot.
If you're happy to use a data storage unit this small, then a UV-EPROM would be very easy to rot. They are still available, online auction sites have them, for not too much money. Even a 27C400, 4 Mbit, is only a few pounds.
Of course they don't store 'files', just raw data, so you'd need some form of external memory controller. Perhaps use an Arduino to address it, and represent its data as a file. FWIW, some CircuitPython modules can be read as USB memory, so could read the data from the EPROM and write a file to their local storage area that represents it.
When I first came across UV-EPROMS early in my career, more decades ago than I care to remember, I was concerned about the recommended write and erase times, so programmed one, then verified it repeatedly, with one minute in the eraser between verifies. It took a few minutes for the first bit to drop out, then most went between 5 and 10 minutes, needing more than 15 minutes to erase the last bit. Quite a range.
Neil_UK's answer made me remember a similar method with more commonly available components. If you're building a system which directly interfaces with a storage device, you might as well just use SRAM, where you fill it with data and then gradually reduce its supply voltage below its recommended minimum. The advantage over EPROM (besides availability of the components) is that it's a lot faster, so you can do a lot more experiments quite quickly to figure out the voltage which produces the most useful cases for you.
I once used this with a microcontroller equipped with internal RAM, which had to detect a power loss and then save some data to non-volatile storage before the capacitors on the board went low enough to cause corruption in the RAM. I wasn't explicitly studying the "memory rot" itself (I was only interested in whether it happened or not), but I did notice that if I took too much time, the corruption seemed to be quite random.
Read the file, flip or set bits randomly according to the profile of error distribution and probability you want, and write the corrupted file back.
Flash loses charge faster at higher temperatures. However, within a week you probably won't be able to trigger that "rot" at temperature where your flash memory (same for magnetization of hard drives, by the way) doesn't combust so....
So, you'll have to go through different routes. As Justme says, you'll have to stress your medium.
The classical stress here would be write stress. A sensible test would:
Depending on your device's quality, your luck, and the speed at which you can write, you should be seeing increasing error rates (i.e. memory cells rotting!) and decreasing read speeds.
Read speed reduction mainly stems from the fact that all modern mass memory employs internal checksums and/or error correcting codes. If they detect a broken memory word, error correction kicks in. Decoding erroneous words takes time, and gets more involved the more broken the codeword is.
The harsh truth is that at modern memory densities, physics isn't nice to anyone, and random bit flips will happen. That's not bad – that's why we have modern channel coding / error correcting codes (they are the same thing). Even a perfectly new storage medium will have some bit errors, but a user will never (or, to be precise, with a probability below some threshold that the user is going to be able to ignore) be subjected to them, because the ability to correct this inevitability of physics is built-in.
By writing repeatedly, you're decreasing the physical qualities of your storage medium more than a week of time could ever do. That simply makes the amount of these physical bit errors (which you don't see) higher. If all goes well, the storage controller is still able to correct these – but it will need to calculate more, and hence take more time, and hence be slower at reading. It might happen that you get more errors than the decoder can correct – and then you actually see a bit error.
It's actually not very trivial measure these, because you'll honestly be measuring the bit error rate of your error-corrected storage medium against the bit error rates of your RAM, which isn't error-corrected (unless you use ECC RAM). That's why step 10 generates small amounts of random data: that will stay in CPU cache, and hopefully not get written to external RAM, which tends to have higher error rates. If you just generated your whole stick's worth of data and wrote it to RAM, and then compared that, you'd be checking your RAM more than your storage medium.
Data retention on flash media depends strongly on temperature, both when written and when stored. The hotter the memory is when you write to it, the longer the data is retained; the colder it is when you store it, the longer the data is retained.
About five years ago, a presentation made the rounds that put some hard numbers on it: for a typical solid-state drive of the time, writing at 25 C and storing at 55 C would give a retention time of about a week. For faster data loss, you can push the temperatures further: write the data while the memory is in a freezer, then store at the highest temperature the datasheet permits.
At some point above you did mention "it should be physical data rot" - but as far as IT systems are concerned, you wrap abstractions into other abstractions and some adapters and some conversions until some utterly software layer gives you bits. Like, you don't physically scribble a rotating hard disk with a magnetic head, and without programming its firmware you likely don't even have a means to.
Also in IT systems, there are deliberately software storage constructs, like RAID arrays inside a computer or block storage over iSCSI and networking. While it may be more apparent with a crashing storage server, note that beside bit-rot on media, each and any layer involved in transporting and interpreting your bits can have errors - all those little processors, memory buffers, firmwares, drivers, cables, connectors etc. involved, as well as their error-detection/correction algorithms applied.
So after such preface, if you are after investigation of errors seen on media by your system, there are error-injection drivers in many OSes that can present a block device or a NIC with the set level of unreliability, which occasionally lies to your higher software layers, stutters, times out, and so you can develop solutions that work well regardless of such errors. In any case, it should not be prohibitively hard to develop one if the OS you work with does not provide such testbeds - for networking it could be an extension of the ubiquitous tun/tap driver (as in openvpn), for storage you can make a loopback device over a file and occasionally sprinkle random bits into that file as the rotting media device.
Cheap writeable optical discs are pretty notorious for rotting relatively quickly (a few years), especially eraseable discs like DVD+RW or DVD-RW. Even write-once discs like CD-R or DVD+R can decay somewhat quickly, depending on the brand and storage conditions. Double-layer writeable DVD discs are even more unreliable in my experience.
Of course, a scratched disc will have have read errors, if you don't want to just wait. Even dust could provoke some, if you don't want to damage the media.
Or leave a disc in the sun for a while, especially dye-side up, especially not behind glass. (Even indoors behind glass should be sufficient; UV from direct unfiltered sunlight is probably not the most important factor. Probably some heating and some actual photon energy doing stuff to the dye chemicals.)
Some burners / readers let you read the raw media (bypassing the 2 layers of error correction), allowing you to see the error rate. So you'll be able to detect rot before it gets to the point of overwhelming the error-correction for a sector. (Audio CDs only use 1 layer of error correction, allowing slightly more space for uncompressed PCM data. Data CDs normally use 2 because a bit-error is more likely to make the whole thing unusable, although apparently the VCD (Video CD) format uses the same mode as audio CD.)
Initial error rate, when you read back a freshly-burned disc, will vary based on how well your burner likes that brand / model of optical disc, and on write speed / strategy selected by the drive's firmware. (There's usually a sweet spot for a given combo of burner and disc, not always the slowest speed but usually not the fastest supported.)
e.g. http://www.digitalfaq.com/guides/media/dvd-tests.htm shows sample output from a DVD-test program, with max vs. average errors per sector, and total, for each level of ECC.
flakeyanderror. \$\endgroup\$