Flash memory and storing data for the long term

I didn’t know this about flash memory:

Flash memory is really cheap. So cheap, in fact, that it’s too good to be true. In reality, all flash memory is riddled with defects — without exception. The illusion of a contiguous, reliable storage media is crafted through sophisticated error correction and bad block management functions. This is the result of a constant arms race between the engineers and mother nature; with every fabrication process shrink, memory becomes cheaper but more unreliable. Likewise, with every generation, the engineers come up with more sophisticated and complicated algorithms to compensate for mother nature’s propensity for entropy and randomness at the atomic scale.

These algorithms are too complicated and too device-specific to be run at the application or OS level, and so it turns out that every flash memory disk ships with a reasonably powerful microcontroller to run a custom set of disk abstraction algorithms. Even the diminutive microSD card contains not one, but at least two chips — a controller, and at least one flash chip (high density cards will stack multiple flash die).

It reinforces the point that we really have no technology for long-term data storage. Hard drives fail, burned CDs and DVDs likewise. Paper is enduring.

Even backup systems like Apple’s Time Machine have problems. If a file gets corrupted on your hard drive, Time Machine will start backing up corrupted copies, eventually over-writing the good ones. What’s really needed is a system that makes a hash of the files to be backed up and stores distinct copies of all modified versions. Of course, that could require a lot more storage space – especially if the files in question are something like videos being edited.

Author: Milan

In the spring of 2005, I graduated from the University of British Columbia with a degree in International Relations and a general focus in the area of environmental politics. In the fall of 2005, I began reading for an M.Phil in IR at Wadham College, Oxford. Outside school, I am very interested in photography, writing, and the outdoors. I am writing this blog to keep in touch with friends and family around the world, provide a more personal view of graduate student life in Oxford, and pass on some lessons I've learned here. View all posts by Milan