Scott Hanselman

On Losing Data and a Family Backup Strategy

March 28, '07 Comments [40] Posted in Musings
Sponsored By

Picture of Hard Drive - Purchased at iStockPhoto.comI lost data today. I'm physically ill about it. I lost all my Dad's email from 2005 and 2006. Here's the worst part...I truly in my heart of hearts don't know how it happened. It's one thing to make a mistake. I make them ALL the time. It another to have something bad happen, play it back in your mind and not know WHY it happened.

As with, I'm sure, all of you Dear Readers, I am the Chief IT Dude for "Team Hanselman." That pretty much means if you have a last name of Hanselman and you're on the West Coast, I'm your IT guy. (Not really, but close.)

My dad's computer was running slowly, and I was starting to suspect he was running out of hard drive space. He had actually complete run out on his little 10 gig C: drive, and his system was thrashing. It was also 88% fragmented. I hadn't setup a scheduled defrag job - something that Windows XP kind of should have done for me, but I digress. As is the tradition, he dropped the computer off last week and I said, "come by next week to pick it up." 

I installed a 20 gig D: drive and moved all his My Documents and what-not on to that drive. He runs Thunderbird (no reason for an Outlook license on his machine) and Thunderbird stores all it's mail in some profile directories deep in Documents and Settings\Application Data blah blah. I made a backup copy of that folder on my Home Server, moved it from his C: to a folder with the new D:\Dave's Documents and told Thunderbird it was over there. No problem, right?

Run Thunderbird and get..."Welcome to Thunderbird!" Ok. Copy the backup back into the C: drive. Run Thunderbird and get..."Welcome to Thunderbird!" Oh, Crap. Rinse, repeat. Repeat until 3am. Open files in Notepad2. Open files in a Hex Editor. Stare in horror at zero byte files. Grep. Pipe. Panic.

I didn't have a good backup plan in place for my Dad's machine. My mom makes CD backups and her My Documents sync'ed to my machine via FolderShare (dangerous because of sync'ed deletes, in fact) but Dad was living footloose and fancy-free and it was my fault.

Having a Family Backup Strategy

Dad was bummed, to be clear, but the first thing he said (after being bummed) was "are the pictures OK?" This simple question hit me hard when I realized that our extended family didn't have a formal backup strategy. Mo and I do, but the Family didn't. I'd be sick if we lost pictures. (Mental note: Blog about the coming end of civilizations that store all knowledge on magnetic media...)

Here's our new strategy. It might seem silly and obvious, but what I learned from this incident is that it's only obvious to the computer guy. You have a plan, a place, I'm sure, on where to meet in case of a fire or emergency. Does your spouse know where The Data is if you don't make it home? Does your spouse have the passwords? Does your sister know what a backup is? I'm learning that drawing up a simple plan like this makes everyone's data life easier. As for my Dad's Thunderbird Email, he's backing it up to My Documents with MozBackup, and everything in My Documents gets backed up. Everything else is backed up into three places: The Home Server at my house, The Rev Drives offsite at the bank, and Mozy.com.

What's your backup strategy? Does it have family-wide visibility and understanding?

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. I am a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by ORCS Web
Wednesday, March 28, 2007 8:28:04 AM UTC
God help us if some sort of freak solar storm erases all magnetic media.
Do you do any encryption of the backups before you send them off-site?
Wednesday, March 28, 2007 8:40:20 AM UTC
Hey Scott,

I have recently been working through the same sorts of issues discussed here. In the longer term I'm looking forward to the "Microsoft Home Server" product (http://www.winsupersite.com/reviews/whs_preview.asp) looks like it is targetted at some of thse very issues.

Anyway, in the end I took the following approach:
1. I have a central server with a dedicated "Backup spindle".
2. Configured SyncBack (great tool) so that it backs up all the data, on all the PC's in my home, directly onto this backup spindle.
3. For my customers, er...relatives...(some on the other side of the planet) I use FolderShare, and while it does sync deletes as well, I make sure they know this, and its a) better than they had before (nothing) and b) I think it beats everything else. I tried SyncBack wih an FTP server for a while but it was just too much effort.
4. I use the encryption features in SyncBack to backup into encrypted .ZIP files with a "Passphrase", that is a LONG password.
5. On my server have another scheduled backup that sends everything up to Amazon S3 using S3 Backup (http://www.maluke.com/s3man/) its in beta but its doing the trick! S3 is just an amazing service, I just checke and 61 gig cost me $9.20 last month.

The way I see it, if I have a localised issue then I can quickly and easily grab the backups off my local server. If it is a more serious issue then I have fail-safe offsite backup with Amazon. I can sleep well.

The final note that I think is important, is that I'm in charge of my encryption, I dont want to just "trust" a third party with this. I didn't see any mention of that in your post, how are you managing the privacy of your data?

Daniel
Wednesday, March 28, 2007 9:23:16 AM UTC
Good to see you're using Mozy, used it for about 6 months and I think it rocks! I especially like the shell integration so you can restore stuff you've deleted right within Explorer.
DuncanS
Wednesday, March 28, 2007 12:56:00 PM UTC
Do you have a Mozy referral code I could use to give you credit for me signing up?

I recently moved and the movers got lost. It was only for an hour, but it got me thinking about all of the data on those servers! Thanks for the reminder - and my condolences on your data loss!
Wednesday, March 28, 2007 1:17:24 PM UTC
Cool article - I only wish that Mozy supported Linux as well! If that was the case then I would sign up this instant.
Bill
Wednesday, March 28, 2007 1:19:33 PM UTC
I 7zip all my source code and upload it to my gmail account using Gspace, that way if my house burns down and I survive I can still complile my precious top-secret apps.
OmegaSupreme
Wednesday, March 28, 2007 1:29:58 PM UTC
I've been using Windows Home Server for the last month, it is a perfectly simple solution to backing up your families computers. I do have to come up with a better offline backup strategy though, going to check out Mozy.
Wednesday, March 28, 2007 1:41:09 PM UTC
Be careful to write down backup encryption passwords (especially for Mozy). You only use them once when you setup the backup and it's easy to forget to store them somewhere you'll remember.
Troy
Wednesday, March 28, 2007 1:50:49 PM UTC
I use S3 - it is extremely cost-effective and I can sleep easy knowing that my data is safe in Jeff Bezos' datacenters :)
Wednesday, March 28, 2007 2:19:32 PM UTC
Scott,

Sorry to hear your dad lost all that email. Have you considered looking into Amazon's S3? You could use Jungle Disk and S3 Fox (Firefox extension) to move your files / backups back and forth. Not sure how it compares to Mozy, but another solution to consider.

You also make a very good point regarding, does your family know where the important files are in case you're not there? I work in banking and one of the things we do in IT is create thumb drives with tools, information, etc and store them in safe deposit boxes across our branches. I'd suggest doing something like this with your spouse.
Wednesday, March 28, 2007 2:26:02 PM UTC
Scott,

I'm sure you already went looking for it, but did you do a full C: scan for the Tbird data file(s)?

Maybe it/they was being stored in another "non-standard" location, instead of under the profile location you expected to find it under.

I did that "accidentally" with my wife's laptop once. We have two profiles on it. One for me and the other for her. I set up Tbird under my account originally and later pointed her profile's Tbird setting to the email data files kept under mine.

I later did a fresh install to upgrade to Tbird 2.beta and forgot that fact and freaked for a while until I remembered where the true location for the mail files was, reassociated them, and was fine again.

Since all it sounds like you did was to move his profile contents back and forth from the C -> D -> again I wouldn't thing that any other files stored elsewhere on the C: were impacted.

Here are some more very good tips from MozillaZine's support you might want to go through: Disappearing Mail: http://kb.mozillazine.org/Disappearing_mail

Also: Since several folks have mentioned backing up passwords...I keep backups of my KeePass password management files locked down inside a TrueCrypt file. Then I feel quite safe stashing copies around on CD/floppy/USB in a few locations. If one gets lost, that's two secure wrappers around my passwords. It's just a pain to update them all periodically.

I did see this on line solution: PassPack ( https://www.passpack.com/ ) which looks interesting, but I haven't signed up to test it yet...and don't know about trusting such an on-line service with the keys to my kingdom....I'm not a security expert so I can't fairly judge...but if it is all they say...it might be another good password backup solution.
Wednesday, March 28, 2007 2:44:38 PM UTC
Data loss (at some point) is inevitable. Losing data sucks. The only fix is multiple backups in multiple places.

I'm not preaching, I'm speaking from experience. I just recovered last night--mostly--from losing our primary drive on the family computer. Thanks to backups I was able to recover everything except for a month's worth of pictures, the most important of which I had "backups" for from sent email or online.

Losing data sucks. It's like death, it's forever. The moment when I realized that my NTFS recovery software (I used R-Studio) wasn't going to recover my pictures I wasn't physically sick, but I don't even know how to describe in words the foul feeling I felt.

Thanks for the tip on Mozy. That was the part I was missing.

-Dan.

p.s. If anyone sees my pictures floating around on the Internet please do me a favor and email them my way.
Dan McCarty
Wednesday, March 28, 2007 2:50:12 PM UTC
I'm using portable thunderbird, so the backup is simple copying of portable thunderbird's directory. With synctoy and winrar set it works just fine ...
a
Wednesday, March 28, 2007 2:51:48 PM UTC
Scott,

Been there...got the grief. I'm sorry to hear you had this unpleasant experience. I ended up finding my files using Stellar to recover the data, although I still have no idea why it disappeared. Even if you didn't erase or format over anything you might try scanning with a data recovery tool just to see if you can find it. Anyway...just my thoughts...good luck.

- Adam
Adam Stocks
Wednesday, March 28, 2007 3:18:54 PM UTC
OmegaSupreme: I read about you on Mozy...

Alternatives to Mozy: Run a cron job of rsync, gzip and mcrypt piped over ssh to your friend's server over his DSL line.

(http://mozy.com/mozy/overview)
Dan McCarty
Wednesday, March 28, 2007 3:24:17 PM UTC
Needless to say I'm now using Mozy ;) However I will still backup the old school way from time to time. Mozy may go bust or something.
OmegaSupreme
Wednesday, March 28, 2007 3:35:54 PM UTC
Aside from the distributed locally and offsite store suggestions a number of you have already mentioned, I also am sensitive to ensure that any user "data" is not stored on the C:\ drive at all. Aside from setting the various paths (HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders) (TweakXP does a nice job of this) to point to a secondary drive, in my case D:\Data.

I do this because I image my machine and regularly restore the image or switch OSs. It takes a bit of time initially to configure it (and certainly discipline when installing any new applications) but for me it pays off. I can see me evolving this more and more to having no local files at all, well ok perhaps locally cached files but web based storage as the point of record and the local cache only for "offline" purposes.

I've been doing a bit of Flex development lately and just started toying with Apollo.. the notion of online storage, offline sync seems to be all the rage (FireFox3, etc)
Wednesday, March 28, 2007 3:53:14 PM UTC
Foldershare creates a "trash" folder where it holds synced deletes. I don't know all the details and rules about what it keeps and when, but there are files in mine.
Wednesday, March 28, 2007 4:01:09 PM UTC
Dag...that sucks...anyhow from personal experience with doing any upgrades of this nature, I have found the best and only solution is to make an exact binary image of the drive in question before doing anything that involves this depth of change to the machine.

How the image is created is moot, as long as you have the ability to extract information from it. Some methods I've used in the past:

1) Norton Ghost (with ghost explorer)
Easiest to set up but costs money to obtain the software, with ghost explorer, extremely easy to extract information from the images...I found this actually to be the easiest way when supporting the inlaws who live several hundred miles away when we'd run into the invevitable problem of "I forgot to have you transfer XXX"...In this case I backed up the data both on an external hard drive that I owned, and also on their machine. In this case it made it pretty easy to walk them through the process of pulling information off the images over the phone (in cases like this my father in law is definately not one who is afraid of "messing up the machine")...

2) Linux Live CD copying /dev/hdXXX over to a file on /dev/hdYYY (can't remember the exact command syntax)
Need to have some understanding of linux, and it is a tad more challenging to obtain the data. But was the fastest way to pull a backup when Ghost was not an option.

3) Manipulation of the hard drive (imaging and what not) to turn the machine into a working Virtual Machine using some of the tools from VMWare
This will be a major hassle to pull off, as there will be some pain that will occur with windows reactivation.

I've found that it was worth the pain however when upgrading a machine from XP to Vista (the machine itself was configured for performing development work--had an instance of Apache + PHP, MY SQL, IIS, and SQL Server...this made it possible to separate the actual upgrade process to vista away from moving the environment (since the VM backup essentially captures not only the data but also the general working environment pretty well)
Arthur Strutzenberg
Wednesday, March 28, 2007 4:35:18 PM UTC
Anyone have experience w/ Memeo - www.memeo.com
CodePoet
Wednesday, March 28, 2007 4:57:41 PM UTC
Did you by chance look at Carbonite? At one of the SD West keynotes, David Platt used it as an example of a very usable interface. It doesn't ask too many questions. Just gets the job done.

Might not be something power users would want, but possibly good for family members.
Wednesday, March 28, 2007 6:45:34 PM UTC
Hi Scott. Sorry to see you've lost data. It's happened to all of us.

I described my backup strategy for the readers of TechBlog earlier this year. I do backup for all the computers at my house, but do not do backups for my family. It's something to think about.

I have a large (1TB in my case) external hard drive that I use for my offsite backups. I would do my offsite backup to an Internet site, but I have hundreds of gigabytes of digital photos and family DV video on my network linux server. It's just too costly to back up that much data over the Internet, but I keep hoping for prices to drop (and bandwidths to increase).
Wednesday, March 28, 2007 7:56:16 PM UTC
Scott,

Doing that for your family is a really great idea. Assuming your family is the "average" user You want it to be as painless as possible for them, what did you do to hook them up and make sure they were backing up the right things? Did you just go over to their house (or use Remote Desktop) and walk them through it? Or did you set everything up and just tell them what they needed to do to make sure their stuff would get backed up?

Thanks!
Wednesday, March 28, 2007 8:01:50 PM UTC
Erik: I tried to set this all up with Remote Assistance, but couldn't get it to go through the firewall. Since my family is 2 hours away, I did this via CoPilot.com - a service that continues to impress. (Be sure to turn Vista to "Window Classic" mode to get decent painting.

JimT: Great article, thanks for the pointer!
Wednesday, March 28, 2007 8:04:23 PM UTC
a: Good point on Portable Thunderbird! That would have made this easier.

Omega: You're right about Mozy...they may well go under, and that's why the encryption and redundancy is important

I use TrueCrypt for local encryption, BTW.
Wednesday, March 28, 2007 8:14:14 PM UTC
Uhm. This makes me think.
Having a backup plan is important and actually doing backups is even more important, but... don't forget to test and practice with restore operations.

Thanks for the good advice.
Fabrizio Camera
Wednesday, March 28, 2007 8:57:45 PM UTC
Hi Scott,

Recently i've had a similar experience where my main HD started giving warning signals that it will soon fail(its still working but according to WD will stop soon). So i started thinking about regular backups because it would really be a disaster if i lose my data(all my work is there). I cannot use any online services because i have about 200GB to backup and i don't have the bandwidth. Plus i don't trust the encryption unless i can encrypt locally and then upload.

So i stared thinking about alternatives for doing this. Maybe using RAID or redundant local backups. Can you offer any advice? I'd appreciate that. I use Linux by the way, so windows only solutions wouldn't work.
Wednesday, March 28, 2007 9:36:14 PM UTC
Mozy is terrific. I switched to them from Carbonite.
Wednesday, March 28, 2007 9:57:06 PM UTC
I tried to sell Chris Brooks on Mozy last time I saw him, but he seemed to prefer Gmail's price point. :)

I wanted to convert my desktop PC at home from Vista to Home Server (beta), but since it's based on Win2K3 server, I doubt the consumer version of Mozy would run on it (they require their pro version for Win2K3). Since Mozy is a requirement for me, and this is the only "always on" machine in the house, that was a deal breaker. It's too bad - I really wanted to try Home Server, and, while its built-in redundancy is swell, off-site backup is more important to me and I don't want to pay Mozy Pro's per-GB rate. So I continue to use that Vista box as my "server" - AKA a glorified file share (a couple of them, actually) running Windows Media Player's media sharing service, and Mozy is constantly backing up the locations on those shares that matter to me.
Wednesday, March 28, 2007 10:29:27 PM UTC
"I made a backup copy of that folder on my Home Server, moved it from his C: to a folder with the new D:\Dave's Documents and told Thunderbird it was over there. No problem, right?

Run Thunderbird and get..."Welcome to Thunderbird!" Ok. Copy the backup back into the C: drive. Run Thunderbird and get..."Welcome to Thunderbird!" Oh, Crap. Rinse, repeat. Repeat until 3am. Open files in Notepad2. Open files in a Hex Editor. Stare in horror at zero byte files. Grep. Pipe. Panic."

I want to make sure I understand the problem. You mean the files on your Home Server are empty? Or do you mean when you copied the files back to your dads computer, the files were empty? Meaning you never actually copied the files over?
Wednesday, March 28, 2007 10:32:22 PM UTC
One last dumb question, have you read the instructions here?
http://www.mozilla.org/support/thunderbird/profile#move
Wednesday, March 28, 2007 10:46:35 PM UTC
mozy.com + foldershare.com = the best free solution
Adam
Thursday, March 29, 2007 10:10:50 PM UTC
Scott

On an off topic I just couldn't help but notice that your diagram uses the same shapes for each computer so I am assuming you didn't draw it by hand. What template did you use to create it. I have an idea for such a diagram.
Thursday, March 29, 2007 11:07:20 PM UTC
The best primary backup, although not free, is a RAID 0 (mirroring) configuration. Most modern motherboards have at least RAID 0 built in, and for those that don't, you can add an IDE or SATA I controller for under $100 (Fry's special). Pop in one or two more matching drives, and you have instant backup with zero management and the easiest restore process imaginable (completely automatic and instant).

Of course RAID only saves you in event of hardware failure. For all other types of data loss or corruption, you still need a last-known-good backup somewhere.

My point is simply that, all of the backup strategies mentioned are great, but they're an amazing amount of extra headache and effort in the event of a simple drive failure.
Friday, March 30, 2007 12:00:47 AM UTC
that's brilliant stuff scott.
Friday, March 30, 2007 1:13:20 AM UTC
Personally I have a network drive here near my primary computer that I backup everything on to prevent data loss through hard drive failure and then I currently use XDrive for backing up all my important stuff again off-site like my CV, old university work, source code and stuff. Will be having a look at Mozy however now as XDrive's web client is clunky and their Windows client is terrible for personal use.
Friday, March 30, 2007 6:28:38 AM UTC
Hi Scott,
did you hear about logmein.com? it's similar to CoPilot.com, it rocks! and they have free plans.
I installed it for family .
Friday, March 30, 2007 12:40:47 PM UTC
Scott,

So do you encrypt your files before you upload them to mozy? . No matter how secure they say they are I don't know if I could trust the files to be out there on the internet somewhere. I also use Truecrypt for local encryption. If however I try to backup the truecrypt volume to mozy then everytime the volume changes it would have to upload the entire thing and can't just do a differential. I'd like to hear you thoughts on this.

Thanks
Sean
Sunday, April 08, 2007 2:09:35 AM UTC
Scott - Yeah, losing files is a stomach-churner -- been there, done that. I wiped all of a vacation's pics off of the camera's SD card. (Found out the "format" option isn't for editing a picture, it reformats the SD ! duh ). Fortunately, we'd just written an 'undelete' program, and I used the beta version and got all the pics back!

Next time, consider using our UndeleteOnClick, or something equivalent to try to recover deleted files. Do it quick before Windows overwrites the space with something new.

Disclosure: I'm with 2BrightSparks Pte Ltd, which produces SyncBack, UndeleteOnClick and other file utilities.

Richard
Friday, April 27, 2007 7:53:33 PM UTC
Mozy is wonderful. I have used it for over a year and it has never failed me.

Mozy has a referral feature where if you enter a current user's code on signup, both get extra space for free. Could someone please use my code? We'll both get an extra 256 megs of room. Here is the link:

https://mozy.com/?ref=LDBPV7

...or just use LDBPV7 as the referral code. Thank you!
Emily
Comments are closed.

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.