9
\$\begingroup\$

I've seen two different pendrives go bad in the exact same way: they were written up to 30-40% of their capacity, producing no RW errors in that part. I was using both pendrives each few weeks/months as "mild" storage for backing up some files. And then one day I decided to copy an additional large file (such as a VM image), filling the pendrive up to 80-90%. And at that moment the new large file would immediately start giving I/O errors.

In the most recent case I calculated a checksum of the problematic file and got a different hash than the one given by the original file generated on my PC. I then deleted the problematic file from the pendrive and copied it over again, and again got I/O errors and bad checksums. However the other old files in the pendrive are still good. Using CHKDSK on the NTFS partition didn't reveal any filesystem error. So it looks like the part of the flash that was empty for a long time somehow suffered memory rot, and the flash controller is unable to detect or mark these blocks as bad.

The second of these pendrives was of a very well known brand. I bought it to replace the old generic one that died with the same symptoms. To avoid problems I decided to buy a quality product, but apparently spending a bit more on good pendrives didn't change the outcome.

So my question is: Is there something about the nature of flash memory that can explain empty blocks going bad after some time in the shelf?

\$\endgroup\$
11
  • 6
    \$\begingroup\$ Probably the flash was always bad and you didn't notice until the controller had to start allocating or reallocating blocks with the huge file write. Unused NAND should not age of unpowered and blank as far as I know. \$\endgroup\$
    – user1850479
    Commented Nov 12 at 13:37
  • 2
    \$\begingroup\$ One point to mention just to make sure there's no conflation of the problem. At least with the no-name brand, it's common for them to (and I don't know how they do this) fake the true size of the drive reported by your OS so they can get away with selling a larger drive. Then when your OS tries to go beyond the actual size, you get errors. If you manually re-format the drive, it should report the correct size afterwards. Might be worth trying this first just in case it's conflating the issue here. \$\endgroup\$
    – Ste Kulov
    Commented Nov 12 at 13:37
  • 6
    \$\begingroup\$ @MisterSmith Formatting does not mean anything. It does not write anything else except the filesystem headers to indicate there are no files. Also getting fake USB sticks from unreputable sellers is a problem. The USB firmware may say they are 4GB and they might not have 4GB of flash. An address pin or second chip with half of the memory may have bad soldering or damaged. \$\endgroup\$
    – Justme
    Commented Nov 12 at 14:00
  • 4
    \$\begingroup\$ @MisterSmith The super-cheap ones don't. \$\endgroup\$
    – Hearth
    Commented Nov 12 at 14:23
  • 2
    \$\begingroup\$ @MisterSmith usually they have to, because a significant amount of the physical pages are going to be damaged and have to be excluded from being used as storage. But that's usually just a firmware function that has to run on factory initialization, once. Afterwards, you know how many of the say 1 Gb of storage you can actually sell, including "write reserve" to help reduce avg. write wear per cell. The lower you declare your storage space, the more reserve, the more robust your product. You can of course declare broken cells as working. You know, to maximize profit regardless of satisfaction. \$\endgroup\$
    – Marcus Müller
    Commented Nov 12 at 15:08

2 Answers 2

Reset to default
20
\$\begingroup\$

Don't discount the possibility of the flash (pen) drive using counterfeit parts and claiming a larger capacity than it actually has. They take the higher address pins and wire them back to the beginning.
A simple byte test will show access, but when you try writing a large amount of data it will overwrite and corrupt the beginning of the storage.

We found this out the hard way when our 256MB flash cards were actually only 64MB. Yes, this was a little while back, around 2010s.

Here is an article about it and method to test flash drives: https://www.passmark.com/support/bit_fake_USB_detection.php

\$\endgroup\$
2
  • 2
    \$\begingroup\$ Wow, this explains so much about why I used to have so much trouble with my cheap micro SD cards getting corrupted all the time \$\endgroup\$
    – tomasz
    Commented Nov 14 at 13:52
  • \$\begingroup\$ I don't believe it was the case. At least not with the last "good brand" pendrive. It was bought original from amazon itself, no 3rd party seller. \$\endgroup\$
    – Mister Smith
    Commented Nov 16 at 16:11
17
\$\begingroup\$

Does unused flash memory degrade faster?

no. The opposite is the case.

So my question is: Is there something about the nature of flash memory that can explain empty blocks going bad after some time in the shelf?

No, but that's also not what is happening: empty blocks usually don't "exist". Let me explain:

between the physical flash storage (i.e., the memory cells in a chip) and you are multiple layers: the software you're using (i.e., your operating system), its filesystem driver, its link (USB?) driver, the host USB controller; a cable; the device USB controller, the link driver, the flash abstraction layer, and then finally the physical connection to the flash memory.

This flash abstraction layer has the job of presenting a constantly unreliable storage device¹ as reliable device.

Thing is: NAND flash (as used in USB drives) can't set a bit from 1 to 0. To do that, it has to erase a whole flash memory page. So, if you change a byte of storage, what happens is that the flash device looks up in a table "oh, there's an unused page, let's freshly erase it and then write the modified data onto it", because that's better than to erase the current data block and overwrite it, simply because that way, you're not erasing/rewriting the same flash cells over and over again for logical blocks that you write to often. And erasing/rewriting is what ages a cell, i.e., increases the probability that a bit is not read as it was written.

That necessitates

  1. the "free blocks" data structure
  2. the "what physical block does this logical block map to right now" table. Or is it marked as unused, i.e., currently not backed by any physical page, and should return all zeros when read?

So, with 2. it becomes clear that you don't have "unused" blocks. You have blocks that have not yet been written to after they've been marked as not in use anymore.

  • So, what probably happens is that your large file copy either was the thing that brought the reserve in "rarely reused" pages down, and so your USB drive had to use pages that have been overwritten many times already, and hence cause storage errors, OR
  • the large copy just heated up your thumb drive a lot, because it needs to do a lot, and maybe your USB port itself isn't great about powering the device, and so the device decided to black out.

¹ this is not a specialty of flash USB drives; it's true for any kind of mass storage or even just transport medium of reasonable density: there's a probability that some bits flip, and that probability is > 0. That's literally just quantum physics at work, right there.

\$\endgroup\$
12
  • 3
    \$\begingroup\$ Some cheaper drives don't seem to have any thermal protection.  I killed a minor-brand USB stick just by writing a large amount of data to it in one go — almost certainly from overheating.  (I was able to write the same amount of data to an identical stick by doing so in chunks with pauses, or by using fan cooling.) \$\endgroup\$
    – gidds
    Commented Nov 12 at 23:14
  • 1
    \$\begingroup\$ @gidds I killed a quite new and useful Sandisk one by writing a Linux live installer to it, and burnt my hand unplugging it, so it can happen quite easily \$\endgroup\$
    – Chris H
    Commented Nov 13 at 10:54
  • 1
    \$\begingroup\$ @gidds: I doubt temperature would be an issue. Flash memory devices can generally write blocks of data much faster than they can erase them, and can only erase memory in rather large chunks (e.g. 256KiB or more at a time). Devices which use flash memory will often use seemingly-idle time to copy live data out of blocks which are mostly full of garbage and then erase those blocks. Writing data too fast to a device with buggy firmware may result in it getting into a situation where it would need to copy data from a mostly-garbage block to a blank region, but there are no blank regions. \$\endgroup\$
    – supercat
    Commented Nov 13 at 18:15
  • 2
    \$\begingroup\$ @supercat That's very interesting — but I'm fairly sure temperature was an issue in my case, at least.  The stick I killed was much too hot to touch, even through the thick plastic shell (that, as Marcus mentioned, may not have had good thermal coupling to the ICs).  When it died, the stick didn't simply go read-only — a known failure mode when the flash has worn out — but vanished and completely refused to mount thereafter.  Whereas writing much more slowly kept the temperature of a similar stick well down; and directing a fan onto it allowed fast writes with no damage. \$\endgroup\$
    – gidds
    Commented Nov 13 at 18:29
  • 1
    \$\begingroup\$ One of the symptoms of "trying to program 1s on top of 0s" (a logic error, only the erase operations can change a 0 to a 1) is excessive current, which also means increased power consumption and heat. First project I was ever given as a NASA summer intern was investigating why a certain data logging device had unexpected current spikes. \$\endgroup\$
    – Ben Voigt
    Commented Nov 13 at 19:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.