On Losing Data and a Family Backup Strategy
I lost data today. I'm physically ill about it. I lost all my Dad's email from 2005 and 2006. Here's the worst part...I truly in my heart of hearts don't know how it happened. It's one thing to make a mistake. I make them ALL the time. It another to have something bad happen, play it back in your mind and not know WHY it happened.
As with, I'm sure, all of you Dear Readers, I am the Chief IT Dude for "Team Hanselman." That pretty much means if you have a last name of Hanselman and you're on the West Coast, I'm your IT guy. (Not really, but close.)
My dad's computer was running slowly, and I was starting to suspect he was running out of hard drive space. He had actually complete run out on his little 10 gig C: drive, and his system was thrashing. It was also 88% fragmented. I hadn't setup a scheduled defrag job - something that Windows XP kind of should have done for me, but I digress. As is the tradition, he dropped the computer off last week and I said, "come by next week to pick it up."
I installed a 20 gig D: drive and moved all his My Documents and what-not on to that drive. He runs Thunderbird (no reason for an Outlook license on his machine) and Thunderbird stores all it's mail in some profile directories deep in Documents and Settings\Application Data blah blah. I made a backup copy of that folder on my Home Server, moved it from his C: to a folder with the new D:\Dave's Documents and told Thunderbird it was over there. No problem, right?
Run Thunderbird and get..."Welcome to Thunderbird!" Ok. Copy the backup back into the C: drive. Run Thunderbird and get..."Welcome to Thunderbird!" Oh, Crap. Rinse, repeat. Repeat until 3am. Open files in Notepad2. Open files in a Hex Editor. Stare in horror at zero byte files. Grep. Pipe. Panic.
I didn't have a good backup plan in place for my Dad's machine. My mom makes CD backups and her My Documents sync'ed to my machine via FolderShare (dangerous because of sync'ed deletes, in fact) but Dad was living footloose and fancy-free and it was my fault.
Having a Family Backup Strategy
Dad was bummed, to be clear, but the first thing he said (after being bummed) was "are the pictures OK?" This simple question hit me hard when I realized that our extended family didn't have a formal backup strategy. Mo and I do, but the Family didn't. I'd be sick if we lost pictures. (Mental note: Blog about the coming end of civilizations that store all knowledge on magnetic media...)
Here's our new strategy. It might seem silly and obvious, but what I learned from this incident is that it's only obvious to the computer guy. You have a plan, a place, I'm sure, on where to meet in case of a fire or emergency. Does your spouse know where The Data is if you don't make it home? Does your spouse have the passwords? Does your sister know what a backup is? I'm learning that drawing up a simple plan like this makes everyone's data life easier. As for my Dad's Thunderbird Email, he's backing it up to My Documents with MozBackup, and everything in My Documents gets backed up. Everything else is backed up into three places: The Home Server at my house, The Rev Drives offsite at the bank, and Mozy.com.
What's your backup strategy? Does it have family-wide visibility and understanding?
About Scott
Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.
About Newsletter
I have recently been working through the same sorts of issues discussed here. In the longer term I'm looking forward to the "Microsoft Home Server" product (http://www.winsupersite.com/reviews/whs_preview.asp) looks like it is targetted at some of thse very issues.
Anyway, in the end I took the following approach:
1. I have a central server with a dedicated "Backup spindle".
2. Configured SyncBack (great tool) so that it backs up all the data, on all the PC's in my home, directly onto this backup spindle.
3. For my customers, er...relatives...(some on the other side of the planet) I use FolderShare, and while it does sync deletes as well, I make sure they know this, and its a) better than they had before (nothing) and b) I think it beats everything else. I tried SyncBack wih an FTP server for a while but it was just too much effort.
4. I use the encryption features in SyncBack to backup into encrypted .ZIP files with a "Passphrase", that is a LONG password.
5. On my server have another scheduled backup that sends everything up to Amazon S3 using S3 Backup (http://www.maluke.com/s3man/) its in beta but its doing the trick! S3 is just an amazing service, I just checke and 61 gig cost me $9.20 last month.
The way I see it, if I have a localised issue then I can quickly and easily grab the backups off my local server. If it is a more serious issue then I have fail-safe offsite backup with Amazon. I can sleep well.
The final note that I think is important, is that I'm in charge of my encryption, I dont want to just "trust" a third party with this. I didn't see any mention of that in your post, how are you managing the privacy of your data?
Daniel
I recently moved and the movers got lost. It was only for an hour, but it got me thinking about all of the data on those servers! Thanks for the reminder - and my condolences on your data loss!
Sorry to hear your dad lost all that email. Have you considered looking into Amazon's S3? You could use Jungle Disk and S3 Fox (Firefox extension) to move your files / backups back and forth. Not sure how it compares to Mozy, but another solution to consider.
You also make a very good point regarding, does your family know where the important files are in case you're not there? I work in banking and one of the things we do in IT is create thumb drives with tools, information, etc and store them in safe deposit boxes across our branches. I'd suggest doing something like this with your spouse.
I'm sure you already went looking for it, but did you do a full C: scan for the Tbird data file(s)?
Maybe it/they was being stored in another "non-standard" location, instead of under the profile location you expected to find it under.
I did that "accidentally" with my wife's laptop once. We have two profiles on it. One for me and the other for her. I set up Tbird under my account originally and later pointed her profile's Tbird setting to the email data files kept under mine.
I later did a fresh install to upgrade to Tbird 2.beta and forgot that fact and freaked for a while until I remembered where the true location for the mail files was, reassociated them, and was fine again.
Since all it sounds like you did was to move his profile contents back and forth from the C -> D -> again I wouldn't thing that any other files stored elsewhere on the C: were impacted.
Here are some more very good tips from MozillaZine's support you might want to go through: Disappearing Mail: http://kb.mozillazine.org/Disappearing_mail
Also: Since several folks have mentioned backing up passwords...I keep backups of my KeePass password management files locked down inside a TrueCrypt file. Then I feel quite safe stashing copies around on CD/floppy/USB in a few locations. If one gets lost, that's two secure wrappers around my passwords. It's just a pain to update them all periodically.
I did see this on line solution: PassPack ( https://www.passpack.com/ ) which looks interesting, but I haven't signed up to test it yet...and don't know about trusting such an on-line service with the keys to my kingdom....I'm not a security expert so I can't fairly judge...but if it is all they say...it might be another good password backup solution.
I'm not preaching, I'm speaking from experience. I just recovered last night--mostly--from losing our primary drive on the family computer. Thanks to backups I was able to recover everything except for a month's worth of pictures, the most important of which I had "backups" for from sent email or online.
Losing data sucks. It's like death, it's forever. The moment when I realized that my NTFS recovery software (I used R-Studio) wasn't going to recover my pictures I wasn't physically sick, but I don't even know how to describe in words the foul feeling I felt.
Thanks for the tip on Mozy. That was the part I was missing.
-Dan.
p.s. If anyone sees my pictures floating around on the Internet please do me a favor and email them my way.
Been there...got the grief. I'm sorry to hear you had this unpleasant experience. I ended up finding my files using Stellar to recover the data, although I still have no idea why it disappeared. Even if you didn't erase or format over anything you might try scanning with a data recovery tool just to see if you can find it. Anyway...just my thoughts...good luck.
- Adam
Alternatives to Mozy: Run a cron job of rsync, gzip and mcrypt piped over ssh to your friend's server over his DSL line.
(http://mozy.com/mozy/overview)
I do this because I image my machine and regularly restore the image or switch OSs. It takes a bit of time initially to configure it (and certainly discipline when installing any new applications) but for me it pays off. I can see me evolving this more and more to having no local files at all, well ok perhaps locally cached files but web based storage as the point of record and the local cache only for "offline" purposes.
I've been doing a bit of Flex development lately and just started toying with Apollo.. the notion of online storage, offline sync seems to be all the rage (FireFox3, etc)
How the image is created is moot, as long as you have the ability to extract information from it. Some methods I've used in the past:
1) Norton Ghost (with ghost explorer)
Easiest to set up but costs money to obtain the software, with ghost explorer, extremely easy to extract information from the images...I found this actually to be the easiest way when supporting the inlaws who live several hundred miles away when we'd run into the invevitable problem of "I forgot to have you transfer XXX"...In this case I backed up the data both on an external hard drive that I owned, and also on their machine. In this case it made it pretty easy to walk them through the process of pulling information off the images over the phone (in cases like this my father in law is definately not one who is afraid of "messing up the machine")...
2) Linux Live CD copying /dev/hdXXX over to a file on /dev/hdYYY (can't remember the exact command syntax)
Need to have some understanding of linux, and it is a tad more challenging to obtain the data. But was the fastest way to pull a backup when Ghost was not an option.
3) Manipulation of the hard drive (imaging and what not) to turn the machine into a working Virtual Machine using some of the tools from VMWare
This will be a major hassle to pull off, as there will be some pain that will occur with windows reactivation.
I've found that it was worth the pain however when upgrading a machine from XP to Vista (the machine itself was configured for performing development work--had an instance of Apache + PHP, MY SQL, IIS, and SQL Server...this made it possible to separate the actual upgrade process to vista away from moving the environment (since the VM backup essentially captures not only the data but also the general working environment pretty well)
Might not be something power users would want, but possibly good for family members.
I described my backup strategy for the readers of TechBlog earlier this year. I do backup for all the computers at my house, but do not do backups for my family. It's something to think about.
I have a large (1TB in my case) external hard drive that I use for my offsite backups. I would do my offsite backup to an Internet site, but I have hundreds of gigabytes of digital photos and family DV video on my network linux server. It's just too costly to back up that much data over the Internet, but I keep hoping for prices to drop (and bandwidths to increase).
Doing that for your family is a really great idea. Assuming your family is the "average" user You want it to be as painless as possible for them, what did you do to hook them up and make sure they were backing up the right things? Did you just go over to their house (or use Remote Desktop) and walk them through it? Or did you set everything up and just tell them what they needed to do to make sure their stuff would get backed up?
Thanks!
JimT: Great article, thanks for the pointer!
Omega: You're right about Mozy...they may well go under, and that's why the encryption and redundancy is important
I use TrueCrypt for local encryption, BTW.
Having a backup plan is important and actually doing backups is even more important, but... don't forget to test and practice with restore operations.
Thanks for the good advice.
Recently i've had a similar experience where my main HD started giving warning signals that it will soon fail(its still working but according to WD will stop soon). So i started thinking about regular backups because it would really be a disaster if i lose my data(all my work is there). I cannot use any online services because i have about 200GB to backup and i don't have the bandwidth. Plus i don't trust the encryption unless i can encrypt locally and then upload.
So i stared thinking about alternatives for doing this. Maybe using RAID or redundant local backups. Can you offer any advice? I'd appreciate that. I use Linux by the way, so windows only solutions wouldn't work.
I wanted to convert my desktop PC at home from Vista to Home Server (beta), but since it's based on Win2K3 server, I doubt the consumer version of Mozy would run on it (they require their pro version for Win2K3). Since Mozy is a requirement for me, and this is the only "always on" machine in the house, that was a deal breaker. It's too bad - I really wanted to try Home Server, and, while its built-in redundancy is swell, off-site backup is more important to me and I don't want to pay Mozy Pro's per-GB rate. So I continue to use that Vista box as my "server" - AKA a glorified file share (a couple of them, actually) running Windows Media Player's media sharing service, and Mozy is constantly backing up the locations on those shares that matter to me.
Run Thunderbird and get..."Welcome to Thunderbird!" Ok. Copy the backup back into the C: drive. Run Thunderbird and get..."Welcome to Thunderbird!" Oh, Crap. Rinse, repeat. Repeat until 3am. Open files in Notepad2. Open files in a Hex Editor. Stare in horror at zero byte files. Grep. Pipe. Panic."
I want to make sure I understand the problem. You mean the files on your Home Server are empty? Or do you mean when you copied the files back to your dads computer, the files were empty? Meaning you never actually copied the files over?
http://www.mozilla.org/support/thunderbird/profile#move
On an off topic I just couldn't help but notice that your diagram uses the same shapes for each computer so I am assuming you didn't draw it by hand. What template did you use to create it. I have an idea for such a diagram.
Of course RAID only saves you in event of hardware failure. For all other types of data loss or corruption, you still need a last-known-good backup somewhere.
My point is simply that, all of the backup strategies mentioned are great, but they're an amazing amount of extra headache and effort in the event of a simple drive failure.
did you hear about logmein.com? it's similar to CoPilot.com, it rocks! and they have free plans.
I installed it for family .
So do you encrypt your files before you upload them to mozy? . No matter how secure they say they are I don't know if I could trust the files to be out there on the internet somewhere. I also use Truecrypt for local encryption. If however I try to backup the truecrypt volume to mozy then everytime the volume changes it would have to upload the entire thing and can't just do a differential. I'd like to hear you thoughts on this.
Thanks
Sean
Next time, consider using our UndeleteOnClick, or something equivalent to try to recover deleted files. Do it quick before Windows overwrites the space with something new.
Disclosure: I'm with 2BrightSparks Pte Ltd, which produces SyncBack, UndeleteOnClick and other file utilities.
Richard
Mozy has a referral feature where if you enter a current user's code on signup, both get extra space for free. Could someone please use my code? We'll both get an extra 256 megs of room. Here is the link:
https://mozy.com/?ref=LDBPV7
...or just use LDBPV7 as the referral code. Thank you!
Comments are closed.
Do you do any encryption of the backups before you send them off-site?