So my Tuesday night plans (a beer, bad TV, absolutely nothing productive) got hijacked around 9pm when a text came in from Dave, who runs a four-person accounting office out of a strip mall near me and pays me a small monthly retainer to keep his network from catching fire. The text just said "is our email broken, everyone got weird." Twenty minutes later I was on his server, and it turned out email was the least of it.
If you haven't been following it, Microsoft shipped emergency, out-of-band patches on March 2nd for on-premises Exchange Server, covering four vulnerabilities that were already being actively exploited in the wild before the fix even existed. Not "patch this eventually," but "someone is already inside servers running unpatched Exchange, right now, today." The group behind it has been getting called Hafnium in most of the writeups. Whether Dave's little Exchange box was actually compromised or just got swept up in the mass scanning that's apparently happening to basically every exposed Exchange server on the internet right now, I genuinely don't know yet. I'm still going through logs at 1am typing this, half paranoid, checking for webshells in folders I never thought I'd have to think about again.
Here's the part that gets me though, and its not really about the exploit itself. Its about why a four-person accounting firm is running its own Exchange server in a closet in 2021 at all. Dave inherited this setup from whoever set it up his office a decade ago, back when "the cloud" sounded sketchy to a guy who does taxes for a living, and nobody's ever had a reason to touch it since. It just sits there, humming, mostly ignored, until a night like tonight when it turns out that thing you forgot existed is also a thing that requires you to know CVE numbers off the top of your head at midnight. I had four browser tabs open comparing CVE-2021-26855 against three other CVEs from the same disclosure, trying to figure out which patch level his exact build was even eligible for, because of course he was two cumulative updates behind.
I'm not trying to make this a "everyone should move to the cloud" post, because I don't actually think that's the universal answer people online are pretending it is tonight. Plenty of small offices have very good reasons to keep mail in house, and plenty of "move to Microsoft 365 and never worry again" advice comes from people who've never had to explain a $180-a-month-per-user line item to a guy who bills $75 an hour. But there is something a little absurd about a filing cabinet of client social security numbers sitting behind a server that hasn't been rebooted since before the pandemic started, patched by a guy (me) doing this on a laptop balanced on a filing cabinet because the "server room" is also where they keep the paper towels.
Anyway. Patches are applied. I ran the script Microsoft put out to check for known indicators of compromise, came back clean, for now, which is the most anyone in this situation gets to say with confidence tonight. I told Dave to change every password on that domain tomorrow morning regardless, admin included, and he groaned about it the way clients always groan about the ten extra minutes it takes, like I'm the inconvenience here and not the four zero-days.
What's sticking with me isn't the specific exploit, it's how fast this went from "obscure CVE disclosure" to "every sysadmin I follow on Twitter is awake at midnight doing exactly what I just did." That's usually the tell that something's bad. The quiet ones don't get everyone moving simultaneously like this.
One small unrelated complaint before I go make coffee I definitely shouldn't drink at 1am: whoever decided Exchange admin center needed to take eleven seconds to load a single mailbox properties page should have to use dial-up for a week. I timed it. Eleven seconds. On a server with basically nothing else running on it. That's not a Hafnium problem, that's just Exchange being Exchange, and it was making an already bad night feel about forty minutes longer than it needed to be.
Going to go check those logs one more time before bed.