Scott Hanselman

Is your stuff backed up? Recovering from a hardware failure

July 7, '16 Comments [30] Posted in Musings
Sponsored By

My computer died and I am sadI had a massive hardware failure over the holiday weekend. I can only tell you what I think happened. I'm not an electrical engineer so if you know more (and I'm sure you do), do feel free to share your thoughts in the comments.

For my main machine in my home office I'm running the Ultimate PC that Jeff Atwood was so kind to build for me a few years back. It's not the Ultimate anymore, but I will say it's still VERY competent. I've upgraded the memory, video card, and SSD (more on the SSD in a moment) and it's a great machine and very very fast.

The tower was connected to an APC 850VA UPS that had worked nicely for a while. I replace just the batteries in my UPS's every 18-24 months. This last week the UPS started a low "scream" so I went over to check it out. It had turned off suddenly, so the PC and all the accessories were also off. I turned it back on and there was a series of snaps and pops, another screen, followed by a non-trivial amount of smoke and a burning electrical smell.

When it was all over, the main fuse of the house had popped, the UPS was dead and smoking, and the power supply in my computer was dead and smoking. Ugh.

I headed down to the local electronics shop and bought a new 1000W power supply, a new APC BR1000G, and went to town, rebuilding my machine. After redoing all the cables and stuff...it didn't boot. I didn't even see the hard drive (SSD). The drive is a Crucial c300. I loved this drive and it worked great for like 6 years...and now it's dead. Turns out that these Crucial drives are known to die when they lose power quickly. I tried to bring it back to live using all the various forums and whitepapers about this known issue, but nope. It's dead.

OK, so have I lost data? What now? Fortunately I backup my systems. And I hope you, Dear Reader, also backup as well.

Stop reading this now and please, think about your backups. Do you have one? Have you tested to see if you can restore from your backups?

Backups always succeed. It's restores that fail. Test your backups by restoring from them.

I've got a number of backups because I practice the Backup Rule of Three.

  • 3 copies of anything you care about - Two isn't enough if it's important.
  • 2 different formats - Example: Dropbox+DVDs or Hard Drive+Memory Stick or CD+Crash Plan, or more
  • 1 off-site backup - If the house burns down, how will you get your memories back?

Here's what my backup situation is/was and how I restored.

While you can use Imaging Software and restore an entire image of Windows or Mac, I find that reinstalling Windows takes less than an hour. I keep a bootable USB key of Windows 10 around. You can also download an ISO and make a USB key quickly. You don't usually need an activation key if you're reinstalling Windows. In my case, I installed the new drive, booted off the USB, signed into Windows with my Live ID (Microsoft Account) and it picked up my Windows license already.

Windows File History

I have a 4TB external drive on my desk that uses Windows 10 File History. This is like the Mac Time Machine feature. It's one of the best little "hidden" features of Windows 10 and everyone should use it. It's actually been around for years. My Documents, Desktop, and any other folders I want are automatically backed up as often as I want. I have a backup going every 30 min and I never think about it. It just works, and I don't notice any performance issues.

In this case, I *did* have crap on my desktop that wasn't in Dropbox and wasn't yet backed up to the cloud. I just hooked up the drive and restored from File History. I literally lost nothing. All my desktop crap was restored in place. If you have an external drive that you always have hooked up but it's not really getting use, setup File History in just minutes.

Multiple Cloud-based Backups

I have a number of clouds in my backup rotation:

  • GitHub - I have github repos, both private and public for code.
  • DropBox - My primary cloud files backup
  • OneDrive for Business - My work cloud files backup
  • Synology - I love my Synology. It's a complete home NAS Server with massive storage, RAID, VPN, Docker, and so much more. A daily joy and a local cloud.
  • CrashPlan - I keep TBs up there and pay them happily for the service.

Related Links

Here's some additional reading on ways to back up your system. Please do also help non-technical relative back up their stuff as well. Every week I hear about someone working on their PhD thesis losing their whole life's work an instant. Backup is a system and it CAN be automatic.

What do you do for backup?


Sponsor: Many thanks to my friends at Stackify for sponsoring the feed this week! it’s what being a developer is all about so do it the best you can. That’s why Stackify built Prefix. No .NET profiler is easier or more powerful. You’re 2 clicks and $0 away, so build on! Prefix.io.

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by SherWeb
Thursday, 07 July 2016 20:28:36 UTC
I put all of my source code in either TFS on-line or github. I have any documents I write saving into onedrive and I have three different computers hooked up to my onedrive account so each of them will have a copy of the file as well. Pretty much everything else I will reinstall if a computer dies.
David Parvin
Thursday, 07 July 2016 20:31:06 UTC
Have you ever tried to restore your synology from crashplan? I'm not going to lie, I'm a little scared and don't know a good way to test it.

I've been looking into the Backblaze solution that now has a built in interface for the synology, but it is more of a sync than a true back up. Way easier to setup and uses way less resources on the synology.
Perry
Thursday, 07 July 2016 20:38:07 UTC
Can I set up multiple File Histories? I would love to set my wife up with both a local (i.e. her own) USB drive for history as well as a second network connected drive.
David Williams
Thursday, 07 July 2016 20:53:09 UTC
Also curious how the restore from CrashPlan went? I tried the free computer-to-computer for a while, but the proprietary backup file format scared me that I'd lose access to the application itself for decoding. I feel more secure with known filesystem where one can see individual files (optionally used with known encryption algorithm).
Thursday, 07 July 2016 21:45:58 UTC
I don't play the Mac is better than Windows or vice versa game (I work on both and really like both) but Mac has one killer feature: if you back up (which I agree is a requirement) you back up the entire disk. So much so that you can take a back up to another machine, restore and be back in business, with all applications working, in about an hour, with no human intervention. That is very cool.

In any case, my backup strategy is to backup to a local hard disk, to keep my important files (nearly all my files!) on Dropbox and to use Backblaze for formal off site continuous backup of my disk.

Thanks

jesse
Thursday, 07 July 2016 22:10:07 UTC
I have a Synology 1815+ which I use as my main storage. I have it set up with 4*Gbit in LACP and use ISCSI drives.
I have another Synology stored in my garage (about 20 meters from my house)
Syno1 backs up to Syno2 with smart recycle (1 per hour for a day, 1 per day for a month, older than a month it will keep a weekly version)
Both devices run on BTRFS so I am able to create a snapshot at filesystem level as well.
This is scheduled once a day and is really fast.
This should reduce the risk for a cryptolocker to lock your files (and backups!)

This way it's already pretty safe. The backup Synology also backs up irreplaceable files, like photos, to another Synology of a friend.
Ronald
Thursday, 07 July 2016 22:42:29 UTC
I recently found that setting up Storage Spaces or using the Windows software RAID mirror can be useful for getting in front of dead drives. It's not a substitute for offsite backups, but if you have an older hard drive that will inevitably fail, it's cheap insurance.

Buy a bigger drive for the mirror and you'll have something nicer to migrate too after failure as well.
Kevin Deenanauth
Friday, 08 July 2016 01:17:32 UTC
1) Backblaze on all the machines here - first backup takes awhile - but it's a good price once it's sync'd up the first time it works great.

2) Time machine to a local drive

3) Dropbox - in theory Backblaze should take care of it - but by using Dropbox as well I get the really important stuff sync'd across my desktop/laptop and online
Paolo
Friday, 08 July 2016 02:09:15 UTC
Google Drive is am awesome solution. You can install the client on multiple machines so you have local copies everywhere in addition to the cloud. Currently I even have the client pointing at an external drive attached to one of my machines that has limited hard drive space.

It is cheap to 3 bucks a month for 100 GB.
Friday, 08 July 2016 07:00:26 UTC
I have a 2TB external drive with FileHistory running to it. All my music, pics and documents folders live in my OneDrive so are auto-sync'd. My local SVN repo is in my docs folder and gets shoved up too.

I currently have a free Office 365 sub so I'm paying nothing for 1TB one drive and getting Office too. I was paying £1 a month for 50GB on top of my grandfathered free space giving me a total of 90GB previously.

I intended to re-sub to the family plan at the end of that since even with just me and my hubby using it, split the bill 50-50 that's only £3.33 a month for 1TB + Office on 3 devices each, that's a no brainer.
Peter
Friday, 08 July 2016 07:45:27 UTC
Windows File History was an exciting new feature with Windows 8 but unfortunately it didn't work out so well for me. It would continuously create multiple copies of files that hadn't changed, filling up the backup storage medium. There was a long thread on it at answers.microsoft.com and looking there again it seems people still have the same problem today.
Steve
Friday, 08 July 2016 07:55:58 UTC
Hello,

I'd add in the list syncthing (with its companion synctrayzor). I've a low cost computer running in my office (with a simple ubuntu server on it), that allows me to remotely clone my important folders.
The advantage of syncthing is that it's 100% open source and does not rely on 3rd party backup tool or undocumented format.

hth
Friday, 08 July 2016 08:07:02 UTC
Hi Scott,
I'm really disappointed about File History, because of three main problems:
1) I've added my sencond partition D: to the backup - and the configuration suddenly removed my partition - I'm checking this from time to time, but it's not really acceptable
So File History just makes sence, if you have all your files in your users dirs.

2) File History is NOT telling you, if something goes wrong - it just writes a log in the eventlog, but is that ok for a not techy user?!

3) File History is not using VSS, so there are some files that don't get backuped
Ralph
Friday, 08 July 2016 11:12:55 UTC
I do use File History but it has limitations, the most severe for me being that it does not backup EFS encrypted files.

For me the best built-in file based backup tool for Windows 7 and 10(!) is "Backup and Restore (Windows 7)". Not to be confused with Windows Image Backup!

"Backup and Restore (Windows 7)" is a file based backup tool. This is the backup tool known from Win7. MS had removed it in Win8 and silently reintroduced it with Win10. You find it via searching the control panel for "backup".

You can configure it to backup any files on your machine. I contrast to File History it does support EFS encrypted files. You can schedule it to run at night and it can reliably wake your computer to execute. It does not have the nicest UI but in my experience it is way more robust than all the commercial tools (especially Acronis) I tried.
Friday, 08 July 2016 11:53:49 UTC
For $60 / year (which breaks down to $5/month) Carbonite works really well. It's saved my but more than once.
Scott Blodgett
Friday, 08 July 2016 12:21:58 UTC
I have a combination of OneDrive and Azure Backup that is working well for my personal files, and Github / VSTS for source control.
Friday, 08 July 2016 12:51:58 UTC
I am a fervent observer of Mr. Atwood's International Backup Awareness Day.

My 'master' backup is a 4TB drive in my home server. My desktop is backed up there, as is anything else personal. I have three 4TB external drives (LUKS encrypted) that I swap every week or two between home, office, and parent's house (less often, 150 miles away).

For work, I have a 1TB laptop drive in a USB3 enclosure (Bitlocker encrypted) that has File History going to it. The fear of the unreliable SSD made me do it.

Mr. Hanselman: As of right now, your story finally made me figure out how to use BTRFS scrub, so I'm scrubbing said 'master' backup, and watching it closely.

sudo btrfs scrub start partition
sudo btrfs scrub status partition
Friday, 08 July 2016 14:02:58 UTC
I just restored 300GB from Crashplan after an HDD failure. It is regularly tested on small restores if the kids get on the computer unsupervised...
Jason Learmouth
Friday, 08 July 2016 16:08:33 UTC
I'd be interested in hearing more about family backup strategies. I've got myself and my wife's laptops covered well enough I think, at least the important things. And I used the Crashplan family plan to add on a parent's laptop. But it'd be nice to see some sort of suggested "formal" strategy for managing backup for the non-tech family if I'm playing IT to them and they're unwilling or unable to have fancy things like on-site NAS systems or spare drives laying around.

Another wrinkle to consider is mobile devices - I'll use dropbox as a central source of things accessible on my phone, and Google Photos works I think enough as a backup of photos/videos. Photos sent via text (Textra!) are auto saved to a folder that is also picked up by Google Photos. And I'm relying on the Android/Google restore system to catch main things and figure I'd deal with anything missing by rebuilding. But there's no real cohesive backup plan of sorts that I'm running on my phone. Nevermind that my wife is on iOS so I'm assuming that is relying on the iTunes backup that may or may not be active.
Pete
Saturday, 09 July 2016 10:26:18 UTC
I come from the land of downunder bandwidth, my uploads at 2.5mbit are considered elite unless you're one of the lucky four people on Australia's shiny new NBN rortfest. I use a few external HDDs, at least one of which is always sitting in a desk in an office kilometres from here. I rotate them when I do a backup, they all sync with the same settings: family photos and OS images. Dropbox has the bare essentials and my application settings, often directly sym-linked on the various machines that use them.

I still use the 2 original Intel X25s I got when they were first released. I have killed exactly one SSD (I have a dozen) and more than a few spinning rust platters in the last decade so don't be frightened of your silent partner. Don't be complacent either, they give you no warnings, no "click of death".
Saturday, 09 July 2016 15:49:24 UTC
Interesting, after seeing a few of these sorts of incidents I'm wondering if a UPS creates more problems than it solves for most people.
Saturday, 09 July 2016 16:16:11 UTC
Thanks Scott, good article as always! I'd like to see followup articles discussing the inevitable intricacies/rat holes such as:
  • File hierarchy. Does each backup location have the same file hierarchy? How do you organise what gets backed up where? Is there one location that's the granddaddy of all others, that has a copy of absolutely everything?
  • De-duplication and discovery. Inevitably you copy files around. How do you find a file that you know you had written, but didn't store in the right place when you did or moved it afterwards? Do you rely on the backup stores automatically de-duplicating for you? Do you ever do spring cleaning?
  • Versioning. Windows File History provides a time versioned store. What happens when you sync this with a flat file backup that's not time versioned, e.g., the Synology? Do these locations have disjoint contents? Do you ever have sync problems, where a newer version of a file is erroneously automatically overwritten by an older file from another backup source?
  • E-mail. Do your backups contain copies of (Outlook) .PST files? How do you reconcile them to maintain a complete e-mail history over time, e.g., you may have deleted some e-mails from newer .PST files, so you have to keep the older .PST files for those e-mails, but that means duplication as well.
  • ...
  • Pēteris Kļaviņš
    Sunday, 10 July 2016 01:04:20 UTC
    Backup rule of there. Very interesting to know. Thanks for nice the post.
    Monday, 11 July 2016 10:31:12 UTC
    As techies, I think we have a responsibility to educate people on backups. I'm a bit concerned about the use of the term 'backup' when referring to store-and-sync services, such as Dropbox (I know there's a store-more-file-history option, but it's paid-for and likely not used by everyone).

    I think we've passed the point where the term 'backup' could have been rescued and kept to mean 'Can go back to any point in time', so perhaps it should be best left meaning 'copy, intended for archival purposes but no guarantees'.

    Can we invent a new term for keeping copies of data with the ability to go back to any point in time?

    "We can't leave for the plane just yet. I'm bunkering my repositories"
    "Did you remember to add the incriminating photos to the datardis?"

    "Is it saved securely?" ... "Yes, bastioned!" ... "That wasn't called for!"




    Monday, 11 July 2016 15:01:05 UTC
    Having a laptop, many times chances are less. Still one should be careful and aware. Thanks for remaniding.
    Tuesday, 12 July 2016 18:52:45 UTC
    DropBox is surely the best solution for cloud files backup. Good to know about your experience. A nice learning indeed. Thanks for sharing.
    Wednesday, 13 July 2016 02:48:16 UTC
    How are people verifying their backups? Say I want to take a system image of my Surface Pro every week. I don't want to restore to my actual Surface from that image every week (to verify), it's my only PC. My first thought was restoring to a VM, but I imagine that's going to fail without the Surface hardware in place?
    Aston
    Wednesday, 13 July 2016 20:29:26 UTC
    Check out last month's discussion for more technical backup solutions: https://news.ycombinator.com/item?id=11817701
    Jed
    Wednesday, 13 July 2016 21:17:35 UTC
    I am trying to reconcile some of these concepts with your recommendation of doing development on VM's.

    I suppose you could keep the vm directory in the list of directories backed up but what does an incremental backup of a vm look like? I also am wondering how you deal with multiple vm's on a single machine? Do you have an SDD for each vm?

    My vm's range from 60GB to 200GB. What am I missing for backing up vm's or do you have an external drive dedicated to each vm?
    Kevin
    Sunday, 17 July 2016 17:12:08 UTC
    How do you store your pictures and videos? I see you have a home server, I have a Synology.
    But how do you handle the files? Are they in the server just as back up? Do you still have all of them in your main PC?

    My laptops drive is getting full and pics and videos are to blame for. Should I just send them all to the Synology and keep the relevant ones? But then, I'd need something to sync them right? Like only keep last month's or last year's pics in the laptop.
    Martín La Rosa
    Comments are closed.

    Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.