The Cloud Has No AC

The Cloud Has No AC

Tech News cloud computing data-centers google cloud outages uk heatwave

So the UK hit 40.3°C yesterday. Coningsby, Lincolnshire, of all places, broke the all-time national heat record by like a mile and a half. And I know a heat record isn't exactly "tech news," but stick with me, because it did something I did not expect: it took down chunks of Google Cloud and Oracle Cloud.

Google's London region (europe-west2, one of the zones specifically) had what they're calling a cooling failure. Not a power failure, not a fiber cut, not some BGP routing mess like we usually get — the actual air conditioning in part of the data center couldn't keep up with the outside temperature, so Google's engineers had to start shutting machines down on purpose to stop them cooking themselves. Oracle's UK South region, also in London, did basically the same thing a few hours later for the same reason. Two of the biggest cloud providers on earth, both beaten by weather.

I love this. I don't mean I love that people's websites went down, or that some poor on-call engineer in Mountain View got paged at 2am their time because a city three thousand miles away was too hot. I mean I love the reminder that "the cloud" is not a cloud. Its a building. Its a building full of metal boxes that generate an obscene amount of heat just sitting there doing math, and that heat has to go somewhere, and normally the somewhere is a very expensive, very redundant air handling system that somebody sized for the hottest day London was expected to have. Except London had a day nobody expected, and the math didn't hold.

There's an old line, I think it's been floating around tech Twitter for years at this point, that "there is no cloud, its just someone else's computer." Yesterday it was more like there is no cloud, its just someone else's computer, and their AC unit is struggling exactly the way yours probably is right now too.

Because let's be honest, this wasn't just a data center problem. The UK doesn't really do home air conditioning. Most houses there were built assuming it never gets properly hot, because historically it mostly didn't. So while Google's engineers were manually powering down racks in Zone A, a huge number of regular people were also just sitting in un-air-conditioned houses sweating through it, no different from their servers. I felt weirdly bad for both groups at the same time.

I'll admit this hit close to home for me for a dumb, specific reason. Years ago — this was maybe 2014, I was doing some contract sysadmin work for a small company that had a "server room" that was really just a converted supply closet with a window unit AC duct-taped to a hole someone cut in the wall. One July, that window unit died on a Friday afternoon. I got the call at home, drove back in, and spent about ninety minutes standing in a closet that felt like a sauna, manually shutting down every non-critical box while a box fan blew across an open server case like that was going to save anything. It did not really save anything. We lost a RAID controller that weekend. So when I read "cooling related failure" in a Google incident report, my whole body remembers that closet. Multiply my one dead window unit by however many thousands of servers Google runs in London and you get yesterday.

What strikes me is how not-redundant this makes the whole thing feel, at least for a day. We talk about cloud infrastructure like its this impossibly resilient, distributed, self-healing thing that doesn't care about a single point of failure because everything's replicated three ways across three availability zones. And that's true, mostly, right up until the failure mode is "the outside air itself is now hotter than the spec sheet allowed for," because that doesn't respect your redundancy plan. It just shows up in every zone in the region at roughly the same time.

I don't think this is some huge indictment of Google or Oracle specifically, to be clear. Building infrastructure for the hottest day in a country's recorded history is a genuinely hard design problem, and 40.3°C in a place that averages maybe 22 in July is a wild outlier by any measure. But it's a useful little data point for anyone who still pictures "the cloud" as some abstract, weatherproof thing floating above the actual planet. It isn't. It's in a building. The building has a thermostat. And this week the thermostat lost.