Of the two - which is the better archival media - Standard HDD or Flash

Question

There are two totally different "bit rot" mechanisms in place here in the two technologies, so which would be expected to last longer? Which is inherently superior for archival purposes. Will a magnetic domain (GMMR) in a "standard HDD" last longer than the floating gate of Flash cell in a CMOS process. i.e which physics dictates the decay process that is slower. Assume climate is controlled, and also assume that cost, interface and physical size is not a contraint. This should be judged/argued based upon the relative merits of the bit decay mechanisms/physics ONLY.

Update - 1 day before bounty expires:

Since this site is about design I limited the question very deliberately. If you're designing a product (with a uController etc.) that will need to be idle for long periods then bit rot does come into play - do you know that the system will light up again when needed? i.e. you probably need archival type storage. Do you design-in Flash? or do you then put in a HDD interface? That is the natural first branch of the decision tree. Once you put in a IDE/SATA or similar interface for the HDD then all the other technologies could possibly come into play. But I deliberately limited it to prevent getting off the rails into arguably declining technologies or at least technologies that would limit the scope of applicability.

I also edited the question to limit discussion on interface(s), since I didn't want discussion on relative interface speeds, density etc. A menton of these issues is of course welcome, but as an adjunct to the answer to the main question(s).

So far there has been a very high degree of anecdotal personal reflection. Not at all desired! I will not award a bounty based upon reminiscing. I will call out @Danny-kmack for at least researching some 3rd party verifiable sources i.e. University of Twente *.ppt, (the superuser link falls into the anecdotal category) but then he fails to bring the salient info forward into the post.

More math, more reputable sourcing, NO anecdotes. Bonuses for bringing up Activation energy and how that ties into the Arrhenius equation will earn extra! Details on how magnetic domains fail and what drives them, and the same for how the Fowler-Nordheim (F-N) Tunnelling onto a floating gate leaks back off - is it still F-N?.

Another call out: this time to @kitt-scuzz for the clever idea of building a system that refreshes the flash drive to ensure that the bit rot doesn't eat the data, you're trading off wear-out for longevity. Unfortunately the wandering/Wondering before hand detracts from the good idea.

Note to Mods: - Yes, I will come back and clean up this superfluous stuff here - either once I see better answers or it is clear that that will not arise.

Remember the key part of EE is the ability to analysis and do the math, understand the physics involved.

This certainly can help people in the future for the design of embedded systems that could reside in the field or even for long term products. - that is the hope anyways.

Answer 1

EDIT Well with the edited content of the question I'm still a bit confused what you're hoping to get out of an answer. The short and simple answer to your question is "it depends." There is no such thing as "the ideal archival medium."

The fact that you say that "refreshing a flash drive" is a good idea confuses the issue even more. You're asking us to say only based purely on the physics of floating-gate flash technology and GMR storage technology "which is superior." Well, refreshing a flash drive every time that it gets close to it's retention limit doesn't discuss either of those topics and yet it still sounded like what you wanted to hear.

Flash can hold data for a long time in certain conditions and with certain considerations taken into the manufacturing process which sets parameters in how effectively the floating gate retains the charge it is given. To be honest, I'm not in the semiconductor manufacturing field so I can't describe to you what these parameters are. But the problem is that even if I could tell you "flash has a longer retention period if you doped region X with material Y and made this part of the gate larger," that doesn't seem to help anyone produce a system which retains information for longer.

One of the major points of my answer was to show you that the flash technology used in PIC EEPROM and the flash used in a standard SSD and the flash used in a compact flash all have wildly different characteristics. Despite the fact that the intel 520 SSD is a much more modern piece of technology than a PIC, it only guarantees retention for 1 year. It depends on the manufacturing process and without a lot of information about what is even possible within the manufacturing world I can't begin to do math and numbers to try and show you that it's possible to produce an inherently "better" archival medium than a standard HDD.

GMR may or may not be the best way to store magnetic information on a drive when you're optimizing for data retention. The first commercial drive using the technology was only released in 1997, who knows what the real world retention time on these drives is? The ideas behind modern storage are generally aimed at optimizing density which frequently negatively affects retention time. The whole comment about me being unable to find WD specs on data retention periods was also another point: it's not a current concern in the data world how long data can be safely stored.

When it comes right down to it, when you look at the fact that you're looking at timelines in excess of about 10 years, it becomes a bit odd to bother trying to figure out a clever method to do it. Today, you can relatively cheaply hold 3TB of data on a single 3½ inch drive. In 2002 the first harddrive was produced which could hold 137GB. In 1983 the first 3½ inch drive was released with a capacity of 10MB. In 1956, the first drive capable of holding 5MB of data was the size of two refrigerators.

What I'm getting at here is that every problem can be optimized. Why you are optimizing it, what you are willing to do to reach that goal, and what you're trying to accomplish set the parameters for that optimization. I cannot produce different floating gates than what is commercially available now so I did my best to provide some different answers based on what is commercially available now. I cannot make my own harddrives so I did my best to answer with some searching on what's available now. Both of those things are at the mercy of what other people did to optimize, which as previously mentioned was not data retention time, but data density.

Sorry for how long winded that was, but I hope you get what I need to answer the question?

---

Oh man, this is a question I think about ALL THE TIME. Something I obsess over is duration of media. For some bizarre reason when I was younger I was frustrated by the non-permanence of things, and I sought to find something in which my collection of dumb crap from the internet could last forever. How long are you hoping to archive for? That changes the ballgame completely, but I'll assume the question is for "as long as you can friggin' get it to last."

Note that I also have to ignore the premise of the question because you start getting into extremely dumb scenarios: if you have a set of Samarium-cobalt magnets each the size of a tube of lipstick storing exactly 1 bit (zero is polarized so that north faces one direction through the coil surrounding it and one is polarized the opposite direction), then it will be AWESOME at storing data for an eternity, and I think more effectively than anything you could do with flash, but not very densely packed and extremely expensive.

So to totally sidestep your actual direct question, the best storage medium is probably ink on paper. Not terribly dense, but there are a number of solutions where, if you get a reasonable printer (around 600 dots per inch), and some nice paper, you can fit ~500KB uncompressed on a single 8.5"x11" sheet. Check out this site for an open source implementation. Note that you'll need a scanner which is pretty capable (around 1000dpi non-interlaced) in order to get the data back. I would assume you would want to make it even less dense to get the information to still be readable decades from now.

Back to what exactly you were asking:

Magnetic media has a shelf life of ~10 years before the encoded information is lost because of the continual pull of the Earth's magnetic field. This may be shorter or longer depending on how densely packed the information is, but the rule of thumb 10 years for magnetic data generally holds up well, but I haven't really tried to check it against anything too modern... really, anything within the past 15 years may be worse than this for retention. I know it definitely applies to 5.25" and 3.5" floppy-disks, but modern HDDs are probably much worse. It used to be that the information was encoded as one-bit per magnetically active grain on the surface of the platter in the harddrive. These days, in addition to the grains on the drive being much more fine (and thus more susceptible to external magnetic influence), they also store multiple bits per grain (so north facing up is 00, and north facing down is 11, north to the left is 01 and north to the right is 10, etc). This means that even if they were as resilient as larger grains to the pull of the earth's field, there is a finer line between the states defining the information. That being said, I was unable to find anything from Western Digital which gave a direct shelf-life to the data on it's commercial products.

Flash storage retention gets worse the smaller the flash cell. Currently, the Intel 520 series (the SSD which I own) says that it "meets or exceeds SSD endurance and data retention requirements as specified in the JESD218 specification". Looking up that standard is a pain, so I'll give you the short version: minimum 1 year at 30°C.

So long story short: HDD wins in the density vs time equation. Hands down. The downside is that if you're going for extremely long shelf-life, you'll have to refresh the data periodically (I probably would try and do it once a year), and after a decade or so you'll have to worry about the integrity of the mechanisms inside of a standard hard drive. There are definitely different flash storage technologies, and essentially, the less-dense you're willing to be the longer-lived your data can been retained without migrating. The ultimate (while still being reasonable) archival medium is probably industrial flash technology, but it won't be very dense. See the block below for some more examples of what I'm talking about!

Some other random thoughts:

• If you ever hear about how CDs have a 70 year shelf-life, this is based on the idea that you have an archival-quality CD and you're VERY gentle with it. CDs have a metallic layer which will oxidize if there is even the tiniest breach in the metallic layer bonded to the plastic, which is why most CDs have a shelf-life closer to about 10 years. Even commercially produced CDs suffer from many different issues. Burnable media, as opposed to media created at a factory which presses specific CDs, also uses an organic dye layer which can fade even more quickly (especially under UV, like direct sunlight). I wouldn't trust most burnable CDs to be free of errors past about 2 years if it hasn't been kept in extremely favorable conditions
• If you really, REALLY need to store that information forever though, check out the mid-range and low-end PICs. They have on-board EEPROM which has a listed retention of more than 40 years! This is what makes me assume that there must be some pretty intense flash storage media out there in the wild
• While looking around for some links about drives, I found this interesting one from Western Digital. Their 8GB compact flash drive has an expected functioning life of 324.3 years if you write 135.2GB per day. This seems like it may be the ideal solution in combination with some sort of industrial microcontroller which refreshes the information on a daily basis or something. You'll probably be more hard pressed to figure out how to keep it powered for the next 324 years though (nuclear power cell/beta voltaic?)... I didn't poke around too much for the expected data retention when not in use, it may not be very good.
• Everything fades eventually! If you're going for extremely long shelf life (i.e. your great-great-great-grandchildren on the Mars colony), there is essentially no mechanism for conveying data which will not eventually fade. As an example: do you even have a 5.25" floppy drive any more? If you had a crazy setup to keep your floppies refreshed from 25 years ago, it probably feels quite silly now! Anything really absurd you work to set up now will likely feel just as silly. USB will eventually disappear, SATA, FireWire, eSata, PCI, all of it. The best way to ensure that your information survives is to put it into some format which can be understood without any assistance by a machine. Everyone who sees a photo album will understand it if they can hold it and see the annotations you wrote in pen. Not everyone will get that the CD with your pictures on it is a bunch of personally curated photos when no one has a CD drive.
• If you want to see some crazy engineering to have a clock which lasts forever, check out the 10,000 year clock

Answer 2

Great question!

Assuming ideal conditions, the HDD should last considerably longer (50+ years) than flash memory (~10 years). However, considering it is impossible to achieve these ideal conditions. If you consider REALLY good, but possible to achieve conditions, flash memory will last much longer(~6 years) than magnetic memory (~2 years). To an an extra twist, radiation hardened flash memory can be reliable up to 20 years. If you are a business you use HDDs connected in a RAID configuration and as long as you replace the drives as needed, your data will last forever!

Sources:

• https://superuser.com/questions/284427/how-much-time-until-an-unused-hard-drive-loses-its-data

• http://www.el.utwente.nl/smi/content/education/is/stand_alone_flash_memories.ppt https://web.archive.org/web/20101227125322/http://www.el.utwente.nl/smi/content/education/is/stand_alone_flash_memories.ppt

• me! (an EE with a very small amount of experience with semiconductor physics and radiation-hardened devices)

Answer 3

HDD magnetically imprints the signal to a depth of only a few hundred grains of cobalt. SSD charges Silcon. Both have decay or half life.

I personally feel that burning a CD/DVD is significantly longer, as it mechanically alters the media, without any such half life.

Answer 4

Various factors are involved and you have specified none of them in your question:

1. Data Volume - How big is your data? GB, TB, EB, etc.?
2. Embeddednes - Do you want to store data in embedded systems? Micros or RAID storage systems?
3. Accessibility - Do you want fast access to your data? ns, ms, s, m...days?
4. Cost - How much you are willing to spend on your archive? \$$$$$, $$$$, $$$, $$?
5. Reliability - How reliable do you want your archive to be? mission critical?
6. Longevity - How long do you want to protect your archive? 1 ,5, 10, 15, 20, 100 years?
7. Feasibility - How feasible is your storage to your application? Do you want to store your humidity data logger data for 100 years?
8. Technology Resilience - How resilient your archival method need to be? Archival technology change fast. If you archive today using a tape drive, that tape drive won't be available after 10 years! If you store your data on a certain type of hard drive, that hard drive interface (SAS, FC, SCSI, PATA, SATA, ST-506, ST-412, ESDI, etc.) would be obsolete in next few years!

My short answer: None of those two you mentioned are good for archival purposes as my archive definition may differ than yours.

Flash and HDD serve two different purpose but in todays world they coincide at the point of near-line storage. I have designed near-line storage units involving both of them.

Flash media delivers faster read time but was never meant for archival purposes. Floating poly could lose its state if proper oxide protection is not used on Si. I have worked very closely with a flash team and they struggled to meet the retention time on HTOL.

In embedded world (microcontollers, discrete flash chips, etc), flash's reprogram-ability feature is more beneficial over other embedded storage like OTP, MTP.

In consumer devices, flash's portability (small size) is the key feature. Compare a SD card over mini-dvd or a floppy in a digital cameras.

Flash storage capacity is limited to the Si geometry and available Si technology.

Hard drives are used for mass storage and their storage capacity surpasses that of a flash device. Hard drives can suffer from stiction (http://en.wikipedia.org/wiki/Stiction#Hard_disk_drives). So if you plan to put data on a hard drive, power it down and store it; It may not be a good idea for an archive.

You can use hard drive as long term archival storage; If it is in an always spinning array with ZFS like filesystem. Hard drives don't account for silent data corruption or bit-rot in your question therefore you would have to employ an auto correcting filesystem for archival.

Although this is not what you asked for but I feel like including this to my answer. In my opinion, various magnetic tape technologies (including and like DLT, LTO, etc) are the best way to archive data. Their prices decline as your data volume increases. They have better shock resistance over hard drives. Their shelf life is longer than hard drives. They have not been compared with flash media in terms of data retention and recoverability. For these reasons, this technology is still sold and bought even when random access media is available and cheap enough.

I would also like to point out that the dye based DVD/CD media may not be used for archival purpose, but pressed media may be.

Further reading:

• Magnetic tape data storage
  
• Digital preservation
• Data corruption

Answer 5

Verbatim have a small amount to say on the subject:

http://www.verbatim-europe.co.uk/en_1/page_are-hard-disks-and-usb-sticks-an-alternative-for-storage-or-are-they-threatened-_2_1.html?con=2

The general subject of data archiving and archaeology is interesting and still under development. See e.g. LOIRP. In that situation, the tapes from the 60s survived just fine in shrinkwrap, but finding the right hardware and software to read them is a serious issue.

There's also probably lessons to be learned from the Museum of Computing, who are trying to get an ICL system 25 from the 70s era running. It's not described on that web page and I can't find a proper reference, but I believe they have read back data from these 40 year old disks; however the mechanical reliability of them is now quite poor and they are not intending to run them on a regular basis.

(not an answer really, but not subjective either)

Answer 6

Frame challenge: I'd ignore the specifics of the media, and instead focus on how to actively counteract bit rot, effectively treating long-term archival as a process, not a product.

Pretty much any engineering output comes with a set of conditions attached for proper operation. A bridge will have a maximum weight it can carry, a schedule and method for recurring checks and a design lifetime after which it needs to be renovated or replaced.

We are making a bit of an exception for consumer goods that are not meant to be used in a scenario where failure would be dangerous -- here we accept that the specifications will be violated and the product will eventually fail.

As that is not an option for long-term archival, you need someone to treat it as a process, even if that is hidden from the end users.

Such a process can be assisted, but not replaced, by technology.

You'd store the data in a redundant way that allows you to detect and correct errors introduced by different error sources, such as Forward Error Correction to counteract random bit errors, redundant storage to counteract failure of entire media and distributed storage to counteract natural disasters that affect large areas.

Each of these has a price/performance trade-off, which becomes part of the specification, and has to be signed off on.

The application requirements define the specifics.

For most businesses' continuity requirements, I'd go with two servers in different datacenters, each with a stack of disks in a RAID6 array with scheduled weekly checks, disable remote administration for these servers and have them poll shared directories of data to be archived in regular intervals, have them store several versions of each file, and sound an alarm if the changeset grows too large or if file contents do not follow the expected format (which would be a sign of a ransomware attack).

The decision between disks and SSDs becomes pretty much moot in such a scenario: I have an automated process in place that verifies weekly that the data is recoverable on a bit level, corrects errors automatically as far as possible and indicates required manual action such as replacement of failed media to the operators. This is complemented by a manual process, where operators verify that the LEDs on the case blink in exactly the expected way and the reports generated by the system are actually read.

Both disks and SSDs are expected to either fail quickly or last several years in this scenario, and either exhibit slow decay of single bits that can be detected by checksums and corrected by rewriting or reallocation, or sudden failure that affects an entire drive.