So yesterday afternoon a big chunk of the internet just... stopped having pictures. You've probably already read three other posts about the S3 outage this week, so I'm not going to walk you through the whole timeline again. What I want to talk about is the one detail that made me laugh out loud at my desk, alone, like a crazy person.
Around 12:40pm Eastern, S3 in us-east-1 started throwing elevated error rates. Within maybe twenty minutes my Slack had gone quiet (not "quiet" as in peaceful, "quiet" as in the app itself wouldn't load), Trello boards were blank, Quora was down, IFTTT stopped iffing, and a genuinely huge number of sites that use S3-hosted assets for images, JS, whatever, started showing broken-image icons where their logos used to be. I was mid-way through uploading some photos for a friend's Etsy-adjacent side project and just watched the upload spinner spin for probably fifteen minutes before I gave up and went to make coffee.
Here's the bit that got me though. AWS has a status dashboard, the little green-checkmark grid that's supposed to tell you which services are healthy. During the outage, people started noticing the dashboard itself wasn't updating properly, still showing green across the board while literally everything was on fire. Turns out the health indicator icons on that page are served from, you guessed it, S3. So the system whose entire job is to tell you "hey, S3 is having a bad day" couldn't render the icon that says S3 is having a bad day, because the icon needed S3 to load. AWS ended up having to manually edit the page and post updates elsewhere because their own monitoring tool was a dependent of the thing it was monitoring. I don't think you could design a better piece of irony if you tried.
I'm not writing this to dunk on AWS, for what it's worth. Running infrastructure at that scale is an genuinely different category of hard than anything most of us will ever touch, and outages happen to everyone eventually, this is just the one that happened to be big enough that my mother-in-law texted me asking if "the internet was broken." What actually stuck with me is how much of the web I use every day quietly depends on one region of one company's storage service, and how invisible that dependency is until it isn't. I run this blog off a fairly boring shared host that's been chugging along since 2011 without so much as a hiccup, and there is a small, smug part of me that felt very pleased about that yesterday. Boring infrastructure has its charms.
Early word floating around (nothing official confirmed yet, so take it as rumor) is that this started with someone running a debugging command that took out more capacity than intended, rather than any kind of attack or hardware failure. If that holds up, it's a good reminder that the scariest failure mode in any big system usually isn't a hacker in a hoodie, it's a tired engineer typing a command they've typed a hundred times before, except this time a couple more servers get included in the blast radius than they meant.
Anyway. My advice, for what it's worth: if you're building anything that leans on a single cloud provider's single region for anything remotely important, spend an afternoon this week figuring out what actually breaks if that region has a bad four hours. Not because you need multi-region failover for your hobby project (you probably don't), just so you're not finding out live, at 12:40 on a Tuesday, that your entire app's images live in one bucket in Virginia with no backup plan. I found three things yesterday that I'd never bothered to check. Not fun discoveries to make while everything's already down.