What are some typical SPI NOR flash failure modes?

Question

I am working with SPI connected NOR flash memory and I've noticed that something like 0.01% of devices experience some kind of write failure over their lifetime. We are careful not to exceed the maximum number of erase cycles specified by our chip (100,000) yet still we see some write failures.

After writing data to NOR flash (erasing first as needed of course), our code confirms that the data written matches the data we were intending to write, if it does not match that is what we consider a write failure.

What are typical modes of NOR flash failure? In other words what types/categories of failure are there? Some examples I can imagine are:

• Permanent bit stuck: no matter how many attempts to program/erase a bit at a particular location is stuck as a zero or one forever
• Transient bit stuck: sometimes a bit at particular location will not successfully be programmed/erased but some amount of retrying works
• Permanent block stuck: an entire block all at once fails to be programmable/erasable, its content can no longer be changed at all
• Transient block stuck: an entire block fails to be programmable/erasable but some amount of retrying works
• Transient bus errors: we are using SPI connected NOR flash, could a bus error cause a read or write to transmit some number of incorrect bits such that what is physically present on the flash chip, retrying the read or write will cause it to succeed

Are these all realistic? Are there other failure modes?

Understanding the failure modes will help me design software mitigations such that our devices might handle the failures and work-around them when possible using techniques such as data replication, retries, ECC, etc.

The particular chip in use is W25Q16JV.

Answer 1

Your prospective failure modes neatly fall into two groups:

• storage medium failures where the memory contents are not what was written,

• transient failures where an EMI transient or another problem causes a command's contents to be scrambled.

Those divide further into:

• permanent medium failures where re-erasing and re-writing a block won't always succeed (it may succeed for some bit patterns but not others) - the block is "cursed" or "failed", in spite of some writes apparently succeeding.

• aging data loss, where the memory is functional, and an erase and write cycle can store arbitrary data,

• transient failures in write command data - a read-back easily catches those,

• transient failures in read command data - a second read will remedy that,

• transient failures where the command itself is misinterpreted - this may leave the part in an undesirable state, a RESET command is needed; it is assumed that the memory is always read-protected unless sending a write or erase command

• transient failures in device state - again, a RESET is needed to make the part functional again

Resilient firmware will:

• bring the WP# pin high (writes enabled) only for the duration or a WRITE or ERASE command, so that reads can never be mis-interpreted as writes,
• retry any failed read,
• read-back before erase to be able to identify an ERASE that didn't touch the targeted block,
• read-back after erase,
• read-back after write,
• re-try erases and writes that failed readback in case the command and/or data got corrupted,
• scrub-read the entire memory if an erase or write readback indicates that ERASE or WRITE acted like a NOP - it is likely the block number got corrupted and some other block got messed up,
• keep scrub-reading the memory in the background to catch bit rot,
• escalate to part RESET and back-off to single data rate serial if there are indications that bus is unreliable in QSPI mode,
• differentiate between bus failures (command/data corrupted in flight) and medium failures - this informs the recovery strategies.
• error detection should have very high rate of identifying any number of wrong bits - use hashes rather than CRC, use the MCU's crypto accelerator to hash quickly if available,
• error correction strategies are about the only place where memory failure modes (bit stuck, dead block, internal address line failures, etc.) will inform the degree of redundancy and thus selection of a particular correcting code, as well as the spatial diversity of data across the address space.