So I don't know if you saw this making the rounds last week, but GitLab had about the worst possible weekend a hosting company can have. Short version: an engineer was trying to fix some replication lag on a Tuesday night, ran a command to clear out a directory on what he thought was the secondary database server, and it was actually the primary. Production. The one everybody's repos live on. Gone in a couple seconds, roughly six hours of issues, comments, snippets and user data just vaporized before anyone could hit ctrl-C.
Bad enough on its own. But then it got worse, because when they went to restore from backups they found out they had five different backup and replication mechanisms set up and every single one of them had quietly stopped working weeks earlier. Nobody noticed because nobody was checking. That's the part that stuck with me. Not the rm -rf mistake, everyone's one keystroke away from that on a bad day, but the fact that they had backups in name only. A backup nobody's tested is basically a rumor.
To their credit they handled the aftermath about as well as you can. Instead of hiding it they put up a public Google Doc live-updating the incident as it happened and then did the whole recovery on a YouTube livestream with something like 5,000 people watching, half sympathetic and half just there for the drama, which, fair, I was one of the drama people for a solid hour on a Tuesday night when I should've been asleep. They ended up recovering most of it from a snapshot that happened to be six hours stale, and losing a chunk of stuff permanently for a few thousand projects. Nobody died, nothing catastrophic in the grand scheme, but if you run anything on the internet it's the kind of story that makes you go check your own house before you finish reading it.
Which is exactly what I did, and it was humbling.
What I actually found when I checked mine
This blog has been running since late 2011 and I genuinely could not have told you, with confidence, when the last successful backup of the database happened. I had a cron job set up years ago on the old host that dumped MySQL nightly to a folder, and at some point I switched hosts and never rebuilt it. So for something like fourteen months this thing has been running with zero backup. Fourteen months of posts, comments, the whole archive, one bad disk away from nothing.
I fixed it this weekend and I'll just lay out what I did in case it's useful, because I don't think my old setup was unusual, I think it's the default state for most personal sites.
- Automate the dump, don't trust yourself to remember. A simple cron job running
mysqldumpnightly, piped and gzipped, takes maybe ten minutes to set up and I'd been meaning to do it for over a year. - Get it off the server it's backing up. This is the step everyone skips and it's the one that actually matters. A backup sitting on the same box as the thing it's backing up survives a bad SQL query but not a dead disk or a hosting company going under. I'm using rsync to push the nightly dumps to a cheap Backblaze B2 bucket, which is running me about 40 cents a month for what this blog needs, so cost really isn't the excuse here.
- Actually restore it once. Not hypothetically, actually spin up a throwaway database somewhere and load the dump in. I did this on a local VM Sunday afternoon and it took two tries because my first dump had a permissions issue that would've made it useless. Better to find that out on a boring Sunday than during an actual emergency.
That third step is the one GitLab's postmortem hammered on, and it's the one I'd bet most of you reading this haven't done either. A backup file existing on a drive somewhere isn't the same thing as a backup that works, and the only way to know the difference is to actually try putting it back.
Anyway, none of this is exotic advice. It's the same three steps every sysadmin has been repeating for twenty years and everyone, myself very much included, nods along and then doesn't do it until something like this happens to someone else and gives them the scare for free. If you run a WordPress site, a Ghost blog, a little Django app, whatever, take twenty minutes this week and actually check. Not "I think there's a backup somewhere," actually go look at the file, check the timestamp, and if you're feeling brave, restore it somewhere and see what happens. I'd rather feel a little silly finding out my dump was empty on a Sunday than find out on a night I'm not there to fix it.