So yesterday morning I'm lying in bed, half awake, trying to check something on my phone, and it just... wont load. Then I see the news trickling in. Snapchat's down. Fortnite's down. Alexa cant tell anyone the weather. Coinbase and Robinhood are having a bad time. Even Duolingo, of all things, apparently choked, and somewhere out there a 900-day streak owl was probably having a stroke about it.
Turns out a big chunk of it traced back to Amazon's us-east-1 region, specifically some DNS resolution mess tangled up with DynamoDB. I'm not going to pretend I have the full technical postmortem memorized (Im sure by the time youre reading this there are twelve very detailed writeups from people who actually work on distributed systems for a living), but the short version is: one region, one database service, and suddenly a shocking percentage of the internet you use every day just stops working. Not because those companies are badly run necessarily, just because they all, quietly, depend on the same slice of infrastructure sitting in Virginia.
Ive read enough "multi-region is the answer" hot takes over the years to know thats mostly marketing talk for companies with the budget to actually do it properly, so Im not going to lecture anyone about that. What it actually made me do, sitting there scrolling the outage threads with my coffee going cold, was go check my own dumb little setup. Not because techpad is some critical piece of infrastructure anyone depends on at 7am (its a blog, Im aware), but because "well that couldnt happen to me, Im too small to matter" is exactly the kind of thinking that gets you a very bad afternoon eventually.
Here's roughly what I went through, in case its useful to anyone else running a personal site or a small side project:
Where does your DNS actually live. Not "who registered the domain" but who's actually serving the records when someone types your URL in. If thats the same provider as your hosting, and that provider has a bad day, you dont just lose your site, you lose the ability to even point people somewhere else while you fix it. I keep mine separated on purpose. Honestly this is the boring, unglamorous part of running anything online and its also the part almost nobody checks until its too late.
When did you last actually restore a backup, not just take one. I have backups. Ive had backups for years. But "having backups" and "having tested that the backup actually restores into a working site" are two very different sentences, and I hadnt tested mine since, if Im honest, sometime last year. Ran through it yesterday afternoon just to be sure. Took forty minutes and found one weird permissions issue that would've cost me an hour of panic if I'd hit it during an actual emergency instead of a calm Tuesday practice run.
Do you know your own recovery time, roughly. Not down to the minute, nobody needs a formal SLA for a hobby blog, but do you know if your site being down means "five minutes of me SSHing in" or "I have to wait for a support ticket and also I dont remember my own root password"? Write it down somewhere that isnt on the server itself. I learned that one the hard way years ago when the one place I had my notes was, obviously, the machine that was down.
None of this is exotic advice. Its not even new advice, people have been saying versions of it since long before yesterday's outage. But theres something about watching literal billion-dollar companies get taken out by a regional DNS hiccup that makes the "eh, Ill deal with it eventually" pile in your own head suddenly feel a lot more urgent. For what its worth, I run my own hosting and deployment through Tricknowtech (just a git push and its live, which I still find slightly magical even after using it for a while), and yesterday was a good excuse to actually go poke at the backup and DNS settings there instead of assuming they were fine because they'd always been fine before.
Anyway. If your stuff was down yesterday and you run something small on the side, maybe spend twenty minutes this week doing the boring checklist instead of just being annoyed at Amazon. Amazon will be fine. Youre the one who has to explain to your mom why your site was down for three days.