So Friday happened. If you were anywhere near a computer, an airport, or a television on July 19th you probably already know what I'm talking about, so I'm not going to do the whole "here's what CrowdStrike is and here's a timeline" thing, since every tech outlet on earth already ran that piece three times over the weekend. What I want to write about instead is the part that's been bugging me since I spent about six hours of my Friday afternoon on the phone talking a friend's small accounting office back from the dead.
Quick version for anyone who's been off the grid: CrowdStrike, a security company whose Falcon software runs on a huge number of corporate Windows machines, pushed out a routine content update to its sensor. Something in that update was malformed, the driver choked on it, and because that driver runs in kernel mode — the same privilege level as the Windows kernel itself, not sandboxed off in userland where a crash just kills one app — the whole machine went down with it. Blue screen, reboot, blue screen again, forever. Delta alone canceled something like a couple thousand flights. Sky News went off air. Hospitals pushed appointments. Microsoft said the number of affected machines was north of eight million, and thats almost certainly an undercount of the real disruption once you count downstream stuff like people who couldn't check in for a flight because the airline's kiosk was bricked.
Heres the thing that actually annoys me, and its not "wow, one bad file took down the world," because thats old news if you've been doing this long enough. Its that this file wasn't even a driver update. It was a content update, the kind of definitions file that ships constantly, multiple times a day, with basically no review process because its treated as low-risk data, not code. Except when your driver runs at kernel level and doesn't validate what it's being fed, a "low risk" data file can crash a machine just as hard as bad code can. Thats the actual lesson here, and I don't think its getting enough airtime under all the "biggest IT outage in history" headlines. The fix isnt "review your code more carefully." The fix is "stop treating rapidly-shipped content as inherently safe just because its not technically a binary."
Anyway. My Friday. My buddy Dave runs a four-person accounting office out near the highway, and around 1pm he texted me a photo of three monitors all showing the same recovery screen. None of his machines would stay up long enough to do anything. The actual fix, once you knew it, was almost insultingly manual: boot into Safe Mode, navigate to C:\Windows\System32\drivers\CrowdStrike, delete or rename the one bad .sys-adjacent file (it was named something like C-00000291*.sys), reboot, done. Except two of his three machines had BitLocker enabled, which meant digging up recovery keys that lived in his Microsoft admin account that he could not log into from a working computer because, well, all three of his computers were down. I ended up walking him through pulling the keys from his phone browser while I sat on speakerphone eating a genuinely bad turkey sandwich from the gas station next to my house. Small stuff, not a hospital or an airport, but multiply that scene by every small office in the country that had two or three machines with CrowdStrike on them and you start to get a sense of why the recovery took days instead of hours. Theres no patch you can push remotely when the machine wont boot far enough to receive a patch. Somebody has to be sitting in front of it.
I dont have a tidy moral here. Kernel-level security software is a tradeoff people made a long time ago and mostly forgot about, and this was the bill coming due. If your endpoint protection vendor can push content to ring 0 on every machine you own without a staged rollout, that's worth sitting with for a minute, whether or not you're the one who has to explain BitLocker recovery keys to a very tired accountant on a Friday afternoon.