My phone started buzzing at 6:40 this morning and by the time I actually looked at it there were eleven missed calls and a text from my aunt that just said "computer is broken again, blue screen, please call." I figured it was the usual thing: Windows Update did something dumb overnight, restart it twice and it'll sort itself out. It wasn't the usual thing.
By 8am I had four more people texting me variations of the same problem. My cousin's small accounting office, two systems down. A friend who runs a little embroidery shop out of her garage, her point-of-sale terminal stuck in a boot loop. And then I made the mistake of checking Twitter (I still call it that, sue me) and saw Sky News had gone off air, Delta and United were grounding flights, and supermarket self-checkouts in the UK were showing the same blue screen my aunt's laptop was showing. Turns out a security company called CrowdStrike pushed a bad update to their Falcon sensor, the thing runs at the kernel level on Windows, and it took down machines everywhere basically at once. Banks, airlines, hospitals, GP surgeries that couldn't book prescriptions, the works.
I do a bit of freelance tech support for family and a couple small local businesses, nothing fancy, just the guy people call before they call an actual IT company because I'm cheaper (free, usually, if we're honest). So my Friday, which was supposed to be spent writing up thoughts on a keyboard I've been testing, turned into six straight hours of the exact same fifteen-minute ritual on different machines: boot into Safe Mode, log in as local admin, navigate to C:\Windows\System32\drivers\CrowdStrike, find the file starting with C-00000291 and ending in .sys, delete it, reboot. Repeat.
Doesn't sound so bad written out like that. It was miserable in practice. Getting into Safe Mode on a machine that's crash-looping isn't always one click, you're interrupting the boot three separate times to force it into recovery, and if the machine has BitLocker turned on (my cousin's did, of course it did) you need the recovery key before Windows will even let you into a command prompt to poke around. Spent close to an hour on the phone with her walking through where Microsoft might have stashed that key, because nobody ever writes it down, nobody ever remembers setting it up in the first place. Found it eventually through her Microsoft account on her phone, which was its own small nightmare because her phone was also not cooperating for reasons that had nothing to do with any of this.
What's stuck with me isn't really the outage itself, outages happen, software breaks. It's how much of the day-to-day plumbing of everything — flights, hospitals, my aunt's ability to check her email — apparently runs through the same handful of security vendors, all with the same kind of low-level access, all capable of taking a machine down before Windows even finishes loading its own logo. I don't think that's some grand conspiracy or anything, it's just what happens when an industry standardizes hard on a few vendors because that's easier to sell to enterprise IT departments. But sitting there manually deleting a .sys file off a laptop in my aunt's kitchen because a company most people have never heard of pushed a bad content update, that's a strange thing to be doing on a random Friday.
The actual fix, once you know what it is, takes about ninety seconds per machine. The annoying part was there was no way to push it remotely to any of these systems, since the whole problem was that Windows wouldn't even boot far enough to let you connect. Every single one had to be touched by hand. I keep thinking about the poor IT person at some hospital or airline with four hundred machines like this and no cousin to call for free labor.
Anyway. Six machines fixed, one very late lunch, and I never did finish that keyboard writeup. Maybe this weekend.