I didn't plan on spending my Saturday morning tailing IIS logs on a stranger's Exchange server, but here we are. If you run anything remotely close to IT for small businesses on the side (which, yes, is still a thing I do for three or four clients even though this blog is supposedly about "tech" in the abstract sense), you already know what week this was.
Microsoft dropped emergency, out-of-band patches on March 2nd for four zero-days in Exchange Server: the ProxyLogon stuff, though nobody was calling it that in the first 48 hours, it was just "the Exchange thing." Not the regular Patch Tuesday cadence, an actual unscheduled fire-drill patch, which if you've been doing this long enough you know is Microsoft's way of saying "drop what you're doing." CISA followed up within a day with an emergency directive telling federal agencies to patch or disconnect, full stop. That's not a request tone, that's a your-house-is-on-fire tone.
So that's the backdrop. What I actually want to write about is the dumb, unglamorous part nobody tweets about: the actual patching.
The part where you find out who's exposed
I've got a dentist's office, a small accounting firm, and a nonprofit I do pro bono work for, all still running on-prem Exchange because migrating to O365 always got vetoed as "too expensive" or "we'll do it next year." Next year has apparently arrived early and violently.
Microsoft published a PowerShell script called Test-ProxyLogon.ps1 on their CSS-Exchange GitHub repo, and running it against each server was the first move — it checks IIS logs, the Exchange HttpProxy logs, and some other locations for the specific indicators of compromise that Volexity and others had published. Two of my three clients came back clean, or clean enough that I stopped hyperventilating. The accounting firm did not come back clean. Found a webshell sitting in a spot I won't bother naming here, dropped sometime in the days before the patch even existed publicly, which tells you these vulnerabilities were being used quietly by whoever found them first (Hafnium, per Microsoft's attribution) well before the rest of the internet's opportunists piled on this week.
That's the part that actually rattles me a little. The scramble everyone's doing right now, all the scanning and patching, most of it is chasing the second wave: the copycats who read the same advisory we all did and started spraying the internet the moment the patch went out, because a patch is basically a map of the hole it closes. The first wave had this for weeks, maybe longer, and picked their targets. My accounting client wasn't picked because they're interesting. They were picked because they were reachable and unpatched, which is a much scarier reason.
An opinion, since I have one
I'm just going to say it: running your own Exchange server in 2021 for a six-person accounting office is not a serious business decision anymore, and this week is the proof. I've been gently suggesting the O365 migration to this client for two years and getting the "but we already pay for the servers" response every time. Sunk cost is a hell of a drug. The migration conversation is happening on Monday and I don't expect much more pushback after finding an actual webshell on their box.
That said, I don't think this is purely an on-prem-bad, cloud-good story either — plenty of orgs on hosted Exchange environments got hit too depending on their setup, and Office 365 has had its own rough weeks. It's more that a six-person office with no dedicated IT staff has no business being the one responsible for emergency out-of-band patching on a three-day turnaround. That's a job for people whose whole job is that job.
Anyway. If you're reading this and you've got an Exchange server sitting somewhere that hasn't been touched since the patch dropped, stop reading blog posts and go run that script. I'll be here later, probably still smelling like the dentist's break room coffee, which for the record is somehow worse than the stuff in my own kitchen and I don't understand how that's possible with the exact same brand of pods.