The Problem of Peristance: Storing and Backing up One's Life a Gigabyte at a Time
I had a scare this weekend. Last week I flashed the BIOS on my lovely Intel Motherboard (you remember, the one I got last August for a song) and forgot about it. A few days later I had to reboot for some reason and after the intial BIOS POST...nothing. Just a non-blinking hard drive light. The weird thing is, if I let the box sit for an hour, it boots. But, if there's trouble, it won't boot again until I let it sit.
Needless to say, I was a little concerned as I was trying desperately to burn a series of DVDs from the seven hours of digital video I shot in Africa. I shot all these Digtal 8mm tapes then ripped them to my external firewire drive. It ended up being about 80 gigs of data. When my C: drive didn't boot, it got me a little panicky and I started thinking about storage.
Here's my setup:
- 45 gig Western Digital EIDE 7200RPM C: Drive
- Contains Windows, my desktop and profile. Also Program Files. No data to speak of. This is the SYSTEM drive.
- TWO 20 gig Seagate EIDE 7200 RPMs MIRRORED with a Promise PCI RAID card.
- The RAID card was the shiznit 4 years ago. It's a FastTrak66. I'm sure it's obsolete now, but it's non-certified Windows 2000 Drivers work fine in Windows XP.
- This array contains a 4 gig partition called DATA with all My Documents and Mo's Documents (her My Documents on her machine points here, although she doesn't know it). While I'm not a fan of partitions, I keep this at 4gig as I figure that's a good "working size" and it forces me to fit and backup all of the Family's Documents onto one single-layer DVD. I back this up to a DVD+RW weekly.
- The remaining 16 gig partition is called STORAGE and is random. I consider it secondary but persistant storage. I back this up less frequently, maybe monthly.
- 200 gig Western Digital External Firewire Z: Drive
- This is the media drive. All my Ripped Music is here, all my Video is here, and all my Audible Books are here.
- 75 gig No Name External Firewire Drive
- This drive is largely unused. I have used it for a Photoshop Scratch Disk or for a Virtual Memory Swap file. However, once it freaked out (I'm starting to not trust Firewire) and can't count on it.
Here's the issue. How do I truly back my life up? As we begin to collect all this 'Media' how to we protect it?
What to Backup of my Digital Life?
I have a ReplayTV with 80gigs of storage (80 hours of video) but arguably the whole drive is scratch. If I lost it all, I'd be sad, but hey, it's TV. There will always be more. Certainly I don't need to back it up.
My C: drive? No worries...well, some, but really, I could pave it and start over, as the DATA is on the RAID Array. I back up the RAID array weekly onto one DVD+RW, and put that in an off site location.
But what about the BIG stuff? How do I backup 200 gigs? 50 DVDs? Not feasible. More and more people are starting to backup their lives on Moving Magnetic Media, and I'm starting to think it's just a ploy to sell more $50 120gig drives.
No one outside of the enterprise seems to mention Tape Backup for the home. PC Magazine is much more likely to suggest that the Small Office/Home Office user buy a Network Storage Device (read: another computer with another hard drive) and suggest that it be hidden in the closet.
But if I really care about my data, how can I protect it?
I Can't Keep Everything
A good example is this recent rip of 80 gigs of video.
Digital 8mm Source Tapes -> 80 gigs of AVIs -> Nero Vision DVD Project -> Rendered DVD Image File -> Final DVD
I certainly can't keep 80 gigs of AVIs around, but if I yank them
Digital 8mm Source Tapes -> DELETED -> Nero Vision DVD Project -> Rendered DVD Image File -> Final DVD
What happens if I have to make a change to the DVD? Since the Project is really a series of timestamp "pointers" to the original video, do I just rip the video AGAIN and hope the pointers line up? If I take the advice of the MyLifeBits guy at Microsoft Research, I'd just keep buying Firewire drives and save EVERYTHING.
What do you do to protect your data?
About Scott
Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.
About Newsletter
We use a SCSI Raid 1,0 configuration with a hot backup drive - we can lose a drive and still keep running. For backup, we use a Tandberg SLR100 SCSI tape drive - a nice device. We do an incremental backup every other day, and a full backup once/month with a rotating set of tapes that are stored off-site in a fire proof safe.
The cost of running another machine just to have a backup seems costly (hydro costs are going through the roof in Ontario) so I use a tape backup. My wife and kids stuff easily fit on an 80GB tape.
Your raw videos are already on tape (8mm digital) so just save those. Obviously in a disaster, you'd lose your edits, but in most applications I've seen, you can save back to your digital camcorder tapes. So for the cost of a digital tape (about $5 in Canada) you can have your original stored in your safe deposit box, and use your new tape to store the edits.
Just my two cents.
TomB
I looked at these two .com services, but they are too expensive as I have not only my 20+gigs of "data" but also 100+ gigs of OTHER. These look like they are about $100 a month or more.
the experience was harrowing. I recovered about 80% of my pictures and music, but my raw captures are gone, and I had already re-used some of my 8mm tapes.
"backup to tape" is reasonable, but at the same time, wrong. tapes deteriorate. I don't want to have to recover my wedding video in 5 years, only to find out that the tape had slowly turned to cellophane.
If I were a corporation, i would use deep, wide 0+1 raid for my mission critical data. I would love to do that at home, but the scsi+fiber mesh overhead is a bit much. It would be great if there were a mechanism to create dynamic, ad-hoc, external RAID arrays... as in, I could come home tommorow with a new firewire drive, plug it in (daisy chained...why not?) to my old drive, and just tell it to become part of the array.
is anyone out there working on that?
Here is my strategy. I determine that if a word document needs to be preserved, I print it out and file it. This safeguards me from a hard-drive crash. Next, I have gobs of digital pictures. For the immediate future, I make two burned copies on CD. For the long-haul (the next 60 years) I plan on making prints of the ones I want to keep and making photo albums of each year or so. Then the CDs rotting is not such a big deal. For video, I think keeping the tapes is the best idea. Making DVDs makes sense too, but I would keep the tapes. I would not even think about keeping 80gigs of video file just for one tape. That is way too much.
So my solution is NOT to back up everything. I don't see a reason. Actually the question is "Why do you need to save EVERYTHING".
1) Hard Drive
2) Flash/USB Drive
DVDs, CDs, Tape, Floppy, and a host of other things deteriorate rather rapidly over time. They all 'promise' long lives but has anyone actually studied whether or not a DVD will keep it's storage for 20 years? No. DVDs haven't even been around for 20 years, so that long term testing won't work (which also proves why the hell do people promise something they've never actually tested).
Also one thing I've noted with my previous experience with Tape drives: They use TAR format and all files tend to meld together. To restore a file on the last part of the tape, you had to sit there while it read the entire thing. Tedious, slow, and not always the best alternative. I've luckily rarely had to recover something using the old 8 gb tapes but when I did, it was sheer torture. Maybe tape has gotten better but I highly doubt it. I bet the only thing improved in the whole tape thing is the speed, that it can read more data per second. It'll still probably have to read the whole damned tape.
There are issues with Hard drives in general. They can degrade but they degrade far slower than any other media. They're also typically more durable, even though you wouldn't want to shake one (while you can shake a DVD). You also have to be a little more cautious but any external drive is usually safe in normal every day operation if the enclosure is built correctly.
I have an external 120 gig HD I use to store all of the Symantec Ghost images for the computers here at work. Luckily the images compress rather well. The drive is kept in the original box, not even a nice little carrying case for the enclosure. I'm sure in the box it moves around a bit, yet I haven't noticed any real loss of performance.
For data backups, we're pretty lucky. We only have <1 GB of really important data, which I backup to my home network nightly. 3 mb DSL does wonders on download speed, which will max our T1 during non-business hours to get the zip files. I set it up to run automatically so that it always backs something up. I have to then only remember "Hey this is about to back up" and turn it off if there's some reason for doing so.
You'd do really well I think in looking at a backup program. Why? Incremental backups. Do you really need to burn your entire collection of data? What if only 2% of those files changed from the last time you backed up? You're then wasting all of that time in a full backup, when it makes more sense to do an incremental. This would keep your changes to a minimum.
This way for instance you could keep a master DVD and keep an 'incremental' USB drive or something. 256 mb should be more than enough to keep say last weeks, or last months 'changed data' on it. You couldn't use it to backup the entire umpteen GB, but you can use it to keep the important stuff. This keeps backup times down plus you keep the most important data at your finger tips. You would then update your 'master backup' disc every so often when the incrimental media started to run out. Then you would simply format the USB Drive and start back over.
Now that I think of it, the idea is backwards. You would want the longer life media (USB drive) to store the data that has to sit around longer, and use a more degrading media for the incremental stuff. Because USB drives aren't up passed 1 GB yet, that proves to be a real problem with that overall design. It's still worth a shot though, but coming up with correct backup/restore schemes that will stand the test of time is definately a huge challenge a lot of us face.
Thanks again for reminding me heh, as you've reminded others
Comments are closed.
As a side benifit, a 250GB SATA drive rocks for VPC images.