The Boring Certificate That Broke My Kitchen Tablet

So yesterday, September 30th, a certificate expired. Not a flashy product launch, not a keynote, just a cryptographic root certificate quietly hitting its expiration date at 14:01 UTC, and it managed to break more of the internet than most actual outages do. I love stuff like this. Nobody tweets about it in advance, and then suddenly half of Twitter is a support forum.

The short version: Let's Encrypt, the free certificate authority basically every blog (including this one) uses to get that little padlock in the address bar, has been issuing certs under its own root, ISRG Root X1, since 2015. But for years that root wasn't trusted everywhere yet, so Let's Encrypt cross-signed it with an older, more widely trusted root owned by IdenTrust, called DST Root CA X3. That cross-sign was the safety net. It's what let ancient devices that had never heard of Let's Encrypt still accept Let's Encrypt certificates. That safety net expired yesterday.

If your device is even reasonably modern, none of this touched you. Chrome, Firefox, current iOS and Android, all fine, they've trusted ISRG Root X1 directly for ages now. But anything running old software that only ever learned to trust DST Root CA X3 started throwing certificate errors the second that clock ticked over. And there is a lot of old software still quietly running the world.

I found out about this the fun way, before I'd even read the news, because the tablet I keep mounted under the kitchen cabinet started throwing "your connection is not private" on literally every site. That thing is an old Nexus 7, running an Android version so far out of support I honestly forget what number it is anymore, and its only job is displaying a recipe site while I cook. I stood there at 7am with flour on my hands trying to figure out if I'd been hacked. Turns out no, I'm just cheap and refuse to replace hardware that still technically works.

Android versions below 7.1.1 never got ISRG Root X1 added to their trust store, so a meaningful chunk of older Android phones just lost the ability to talk to any site secured with a Let's Encrypt cert. Same story for old OpenSSL builds (1.0.2 and earlier), old Java installs before 8u141, and macOS before Sierra 10.12.1. People online were reporting real breakage on Sophos and Palo Alto firewalls, QNAP NAS boxes, and a pile of smart-home and point-of-sale gear running embedded Linux that nobody has patched since it left the factory. This is the actual texture of "legacy systems" that people wave their hands about in enterprise slide decks. It's not abstract. It's a firewall admin panel that suddenly won't load, or a card reader at a coffee shop that can't verify a payment, or my kitchen tablet.

Let's Encrypt knew this was coming, obviously, they've been publishing warnings about it since at least 2020. Certbot and most modern server configs already stopped serving the old chain by default months ago specifically to force this issue early, on purpose, rather than let it happen as a surprise. I actually respect that. It's a rare example of an infrastructure provider forcing the pain forward instead of just hoping nobody notices when the real deadline hits.

Anyway, the fix for my tablet situation is either "throw it away and buy a $60 fire tablet" or "manually install the new root cert," and I already know which one I'm too stubborn to do properly, so it's probably going to sit there broken for a month before I deal with it. That's basically the whole lesson of yesterday, honestly: a huge amount of the internet still runs on devices nobody thinks about until the moment they stop working, and then everyone acts shocked that ten-year-old software has ten-year-old problems.

If something on your end broke yesterday and you're trying to figure out why, check whatever's oldest first. It's almost always the oldest thing.