Had one of those weeks where a small thing goes wrong and you realize just how much of your setup was held together with hope. Nothing catastrophic happened, don't worry, the site's fine, you're reading it right now. But my laptop's SSD started throwing SMART errors on Saturday morning, right as I was about to write a post, and it turned into a three-hour detour into "wait, when did I last actually back this thing up."
Answer: April. Not this year. Not even confidently which April.
techpad's been running since late 2011, and in that time I've had exactly one real data scare, the great database corruption of 2016, which some of you might remember because I wrote an embarrassingly long post about it. Ever since then I've had "backups" in the loosest possible sense. A cron job that dumps the database somewhere, a vague memory of also copying the uploads folder at some point, and a lot of faith that none of it would ever actually be tested. That's not a backup strategy, that's a backup superstition.
So this week I actually fixed it, and I'm writing it down partly for you and partly so future me has something to point at instead of re-learning this in another decade.
What I ended up with
Three layers, because one is fragile and I've learned that the hard way twice now.
- A nightly
mysqldumppiped straight into gzip, timestamped, kept for 14 days locally on the VPS. - Those dumps plus the
wp-content/uploadsdirectory get rsynced off-box every night at 3:40am to a cheap object storage bucket. Not because 3:40am is magic, just because it's after the dump finishes and before I'm awake enough to notice if it hiccups. - Once a week, a copy lands on an external drive here at home, because I do not fully trust any single provider with the only copy of fourteen-plus years of writing.
The rsync-to-object-storage step is the one I kept putting off because it sounded like a project. It wasn't. It's a twelve line shell script and a cron entry. The part I actually spent time on was testing restores, which is the part everyone skips and the part that actually matters, a backup you haven't restored from is just a file that exists.
Here's roughly the dump-and-push script, trimmed down:
#!/bin/bash
STAMP=$(date +%Y%m%d)
mysqldump -u techpad -p"$DB_PASS" techpad_db | gzip > /backups/db-$STAMP.sql.gz
rsync -avz /backups/ remote:/backups/techpad/
find /backups -name "*.sql.gz" -mtime +14 -delete
Not exciting. It doesn't need to be. The boring scripts are the ones that are still running in 2030.
The VPS I moved this blog to a couple years back is through Tricknowtech, and honestly the git-push deploy is the reason I bother pushing small fixes like this instead of letting them rot in a todo list, I can tweak the cron setup, push, and it's live in about ten seconds, no separate deploy dance. Doesn't fix laziness entirely, but it removes one excuse.
The part that actually scared me
I did a full restore test into a throwaway subdomain, and it worked, mostly. Posts, comments, images, all there. But three plugins I apparently installed in like 2019 and forgot about broke on restore because they'd been abandoned by their authors and PHP has moved on without them. Nothing important, an old syntax-highlighter plugin and something that used to auto-tweet new posts (RIP that whole use case, incidentally). Deleted both. The blog is faster for it, which is a nice bonus nobody warns you about, half your plugin folder is probably dead weight you're scared to touch until a restore forces the issue.
If you run any kind of personal site, and I'd bet a decent chunk of you reading this do, go check when your last backup actually was, not when the cron job says it ran, when you last confirmed the file is good and restorable. I was confidently wrong about mine for close to a year and a half. The failure mode isn't dramatic, it's just quiet, right up until the one day it isn't.
Anyway. SSD's fine, apparently it was a firmware thing and a BIOS update sorted it. But I'm keeping the new backup setup regardless, because "it was probably fine" is not a sentence I want to be saying about fourteen years of posts.