So this was the week half my RSS reader went dark.
If you were anywhere near Twitter Monday night you already know the broad strokes: Sandy came through, the East Coast got hammered, and a big chunk of lower Manhattan lost power. What I didn't expect was to spend Tuesday afternoon clicking on Gawker, Jezebel, io9, Kotaku, Lifehacker, Deadspin, basically the entire Gawker Media stable, and getting nothing back but a spinning wheel. BuzzFeed was down too. So was a chunk of Huffington Post's stuff. For a while there it felt like someone had unplugged a whole section of the internet, which, it turns out, is more or less exactly what happened.
The story, as far as I can tell from piecing it together over the last couple days, comes down to one building: 75 Broad Street, down in the Financial District. There's a data center in there run by a company called Datagram, and Peer 1 has space in the same building. When the flooding hit, the basement went underwater, and that's where the fuel pumps for the backup diesel generators lived. So you had this almost cruel irony where the generators themselves were fine, sitting up on a higher floor, ready to go, but the pump that was supposed to feed them diesel was sitting in several feet of storm surge. No fuel getting up, no backup power, no matter how good the generator itself was. From what I've read, people were literally forming bucket brigades up flights of stairs trying to keep things running on hand-carried diesel. Somewhere between impressive and completely insane.
I keep coming back to how fragile this all is. We talk about "the cloud" like it's some abstract, weightless thing floating above us, and then a single basement in Manhattan floods and a meaningful slice of the sites I read every day just stops existing for the better part of a day. Gawker actually got a temporary version of the site running off Blogger of all things, which I kind of loved, honestly. There's something almost funny about one of the biggest media companies on the internet falling back on the same free platform a teenager might use for a fan blog, and it working fine as a stopgap.
Small aside, but it's been bugging me: I checked my own host's status page during all this, out of morbid curiosity more than anything, and their "99.9% uptime guarantee" language is doing a lot of work for a company that, as far as I can tell, runs out of a single facility with no real geographic redundancy. I'm not naming names because it's not really about them specifically, it's about how many of us are running sites and small businesses on hosting setups where a bad afternoon in one zip code could take us out completely. I've been meaning to set up an off-site backup of this blog for about a year now and keep not doing it. This is the kind of week that makes you actually go do the boring maintenance task you've been putting off.
There's also just the human side of it that's easy to skip past when you're talking about servers and pumps. People who work at these companies were dealing with actual flooded homes, no subway, no power at their own apartments, and still trying to keep sites up for readers who mostly just wanted their normal Tuesday internet. Marc Andreessen, whose firm has money in some of these companies, was tweeting through it too, which was a weird little window into how differently a storm hits when your job also happens to be the internet.
None of this is a knock on Datagram specifically. Basements flood. Nobody planned for a storm surge that high in that part of the city, and hindsight is very cheap. But it's a pretty clean, small-scale illustration of a bigger problem that gets glossed over whenever someone tells you to just "put it in the cloud" like that solves everything. The cloud is a bunch of specific buildings with specific basements, and sometimes the basement is the whole ballgame.
Election Day is Tuesday, and half of the coverage the next 48 hours is going to be storm recovery versus horse-race politics fighting for the same headlines. I don't have anything smart to say about that part. I just wanted to write down the data center thing while it was fresh, because in six months this is exactly the kind of detail that gets forgotten in favor of the bigger storm story.