18
\$\begingroup\$

I am working on a device that uses the Microchip MDDFS library to store data to an SD card. The logger will log data at a maximum rate of 1 entry (56bytes) every minute. The problem is the device may lose power at any time, potentially in the middle of a write sequence. I am wondering what is the best way to protect my data against corruption. I have found that if the file is open when the power is lost, all data that was written to the file after the last file-close is lost. I don't know if the same holds true if power is lost in the middle of the write sequence.

Since the write procedure doesn't happen very frequently I could open the file, write the data, and then close the file, every time data is logged. Would this approach damage the sd card over time?

Another approach could be to keep the file open but after every 10 or 50 writes I could close the file and then re-open it.

I could also buffer data in memory, then flush the data occasionally maybe after a kbyte or so.

The last idea I had was, in my circuit, I could add a large capacitor that would provide power to my pic/sd card long enough after the power is disconnected to quickly close the file. The problem with this approach is that the time it takes to close the file and/or save data is very inconsistent. From my understanding, this time can very depending on the current place in a flash page that the file is in.

Anyways, what would you guys suggest?

\$\endgroup\$
  • 2
    \$\begingroup\$ If you switched to raw NAND flash with a minimal file system, you could drill through many of the layers of abstraction currently hindering your ability to commit small writes, and likely gain the ability to do partial block writes immediately as the data became available. You would also have protection against an end-user substituting a card with different performance details (maybe even a marginal grey-market one) down the road. \$\endgroup\$ – Chris Stratton Dec 10 '11 at 20:17
16
\$\begingroup\$

A few things can happen when you write data to a file. I'm going to describe the sequence that needs to happen for data to be safe, not necessarily library calls.

When you're writing, and adding on to the end of the file (normal write mode), you read the last block of the file into memory, modify it with your write data, and then write the whole block back to the SD card. If the block is full, then a new block must be found in the File Allocation Table (FAT). After finding a new block, the FAT must be updated, which is a read-modify-write cycle. If we're done with the file, then we need to update the file attributes (such as file length) in the root directory, which causes another read-modify-write cycle.

Minimizing your write time

  • Make sure that the file is already holding your data when you write a sector. If you start out with a large file and overwrite the data instead of appending the data, the data will be safe as soon as the SD card sector write is finished. You can eliminate one to two read-modify-write cycles that way. My start-up code would write 0's to a file in sector increments until the SD card is full, and then rewind to the beginning of the file.

  • Make the size of your data entries such that an integer number of entries fit in a sector. I would bump up your entries to 64 bytes. While this is less efficient, it will prevent you from needing to read-modify-write two sectors.

  • Create a variant of the FSwrite function that allows you to write whole sectors. If you keep the whole sector in SRAM, then your cycle goes from "read-modify-write" to "modify-write"

Keep your PIC and SD power on as long as possible

  • Big capacitors are good. 470uF should give you more than enough power to finish a write cycle.

  • Make sure your power source won't suck the power out of your back-up capacitor! Add a diode if necessary.

Know when you're out of power

  • A big power supply cap will give you 10ms or more to wrap things up with a SD card, but don't press your luck. Use a pin on your microcontroller to see if your power source is still good, and don't start a write if your source is dead.
\$\endgroup\$
  • \$\begingroup\$ Thanks for the info. Very helpful. I'll see what I can do with it... \$\endgroup\$ – PICyourBrain Dec 30 '10 at 19:18
  • \$\begingroup\$ How long do you think it would take to write all zeros to a 4gb card? Seems like a long time. Also, do you have any code examples you could share for modifying the fswrite function to allow whole writes to whole sectors? \$\endgroup\$ – PICyourBrain Jan 3 '11 at 13:04
  • \$\begingroup\$ Also, if you write all zero's to the file. How do you keep track of where the end of your actual data is? Do you just read all the data at the beginning and find the string of zeros? \$\endgroup\$ – PICyourBrain Jan 3 '11 at 13:14
  • 1
    \$\begingroup\$ If you're using FAT16, I believe you're limited to 2GB. I believe the SD/MMC card has an "erase blocks" function, which doesn't appear to be implemented in the MDDFS library. I used a proprietary code library for my SD card project, so I can't share any code examples. In order to find the last data, you would need to read until you found all 0's in a data record. If your data record can be all 0's, I'd advise adding in some non-zero data, or a delimiter of some sorts. \$\endgroup\$ – W5VO Jan 3 '11 at 15:08
6
\$\begingroup\$

One issue not yet mentioned with SD cards (or MMC, CompactFlash, etc.) is that while an SD card may appear to the host as a simple collection of 512-byte sectors which may be read and written in arbitrary order, flash devices generally store 528-byte pages in groups which are 32KB if not larger each, and the only operations supported are to either write to an otherwise-blank page, or to erase an entire group. To deal with this limitation, the controller on an SD card will keep a table which will allow any logical sector to be mapped to any physical page. When a request is made to write a sector, the controller will find a blank page somewhere on the chip and update the mapping with the new address of the sector in question. If blank pages get to be in scarce supply, or at various other times, the controller will move an erasable-group full of pages to new addresses, and then erase that entire group.

The significance of this is that the act of writing to a particular logical sector may require shuffling around the data from many logical sectors. If something goes wrong in that process, it could result in the corruption of any arbitrary sector--not just the sector the card was asked to write. A good SD card controller should be designed to perform the data-shuffling operations in such a way that if power is lost during a data-shuffle, it will be able to figure out which parts of the operation had completed and which had not, and consequently be able to finish off the operation properly. Unfortunately, I have no idea how one can tell whether the $5 SD card one picked up at a discount store will be any good in that regard.

To be sure, even if an SD card is absolutely perfect from the standpoint of ensuring that every write operation which was reported as having been completed will, in fact, survive a power failure (i.e. ensuring that whether or not all of the work the write would have caused is complete, enough has completed that the card will finish the operation when power is reapplied) that doesn't mean the host operating system won't have trouble if it performs some but not all of the data writes it intends. Nonetheless, it is important to bear in mind that if the SD card can't uphold its end of the "bargain", there's nothing that can be done on the host side software to prevent data loss from power failure.

\$\endgroup\$
  • \$\begingroup\$ That is a very insightful comment. \$\endgroup\$ – fred basset Jun 6 '14 at 3:18
5
\$\begingroup\$

I would also suggest using some sort of checksum to verify the data on the SD is correct whenever it needs to be read.

\$\endgroup\$
3
\$\begingroup\$

Perhaps this supercapacitor at Sparkfun would solve the problem.

\$\endgroup\$
  • 2
    \$\begingroup\$ It would hold the memory of the PIC, but it has a 10uA max current rating. I don't think you could write to a SD card on that much current. \$\endgroup\$ – W5VO Dec 30 '10 at 17:14
  • 1
    \$\begingroup\$ The concept is good though. A quick search found Illinois Capacitor (illinoiscapacitor.com) has supercapacitor up to 8 F and able to support currents up to 4 A. Adding one of these, or a battery, would give the micro a chance to finish a write and shut things down nicely if the main power source failed. \$\endgroup\$ – The Photon Dec 11 '11 at 3:10
3
\$\begingroup\$

Since the write procedure doesn't happen very frequently I could open the file, write the data, and then close the file, every time data is logged. Would this approach damage the sd card over time?

As with any engineering problem, you'll need to deal with tradeoffs here.

Is it critical that no-data be lost at all? Then I'd do the above. You'd have more damage done by losing the data than by ruining a card. You may want to do a stress test, of sorts, to determine how many times you could perform that operation before the card becomes corrupted. If your'e cool with the length of time it took before the card became unusuable, and it seems to be an acceptable length of time before changing the card, I'd go with that route.

\$\endgroup\$
1
\$\begingroup\$

If you want just to store data there is no need to a file system. the write operation will be done directly over the SPI by selecting the block adress. by doing that you minimize the write time and you the risk of data corruption.

Even in case of power loss and no luck, you will lose just one entry (which maybe acceptable on some system).

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.