The Bash Bug Ate My Weekend

The Bash Bug Ate My Weekend

Personal bash linux security shellshock sysadmin

So that's my weekend gone.

By now everybody with a terminal window open has heard about Shellshock, the Bash bug that dropped on Wednesday and immediately got compared to Heartbleed by every outlet that wanted the traffic. I rolled my eyes at the comparison at first — Heartbleed took months to really sink in for most people, this thing had a one-line proof of concept you could paste into any terminal and watch your own machine confess:

env x='() { :;}; echo vulnerable' bash -c "echo this is a test"

If your box prints "vulnerable" before it gets to the actual test line, congratulations, your shell has been happily executing whatever code was hiding after a function definition in an environment variable since approximately forever. Turns out this bug has been sitting in Bash since 1989. Twenty-five years. I was two years old when this thing was introduced and it just now got noticed.

I didn't think much of it Wednesday night. Then Thursday the second CVE showed up because the first patch was incomplete, and by Friday there was a third one. At that point I stopped ignoring it and actually looked at what I've got running.

Here's the annoying part nobody warns you about: it's not one server, it's every server, plus the router, plus the NAS in the closet, plus that little DigitalOcean droplet I spun up two years ago to host a Node thing I never finished and forgot to ever shut down. I spent a chunk of Saturday just making a list. Turns out I had four boxes touching the internet in some way and I only remembered three of them off the top of my head.

The actual patching part is boring, which I mean as a compliment. apt-get update && apt-get install --only-upgrade bash on the Ubuntu machines, done in about ninety seconds each. The pfSense box needed a manual package update through its own interface, which took longer only because I'd forgotten the admin password and had to dig through an old text file to find it (yes, I know, I need a real password manager, I've been saying that since 2012 and never done it). The one that actually worried me was this blog's host — it's sitting on Apache with a couple of ancient CGI scripts I inherited years ago and never fully understood, and CGI is exactly the kind of thing Shellshock loves, since it hands attacker-controlled headers straight into environment variables that Bash then processes. I patched Bash itself in about a minute. I spent closer to two hours afterward just staring at those old CGI scripts trying to decide if I trusted them enough to leave alone. I did not, in the end, trust them. Two of them are gone now and nothing broke, which either means they were dead weight or I'm about to find out the hard way that they weren't.

My father-in-law texted me Sunday asking if his little WordPress site was "going to get hacked by the bash thing," and honestly I didn't have a great answer for him beyond "probably not, but let's check anyway." Most shared hosting isn't directly exposed the way a self-managed box with custom CGI is. But "probably not" isn't the kind of sentence that makes anyone feel better, so I just ssh'd in, ran the one-liner, saw it was already patched by his host, and told him he was fine. He seemed disappointed there wasn't more drama.

What actually gets me about this one isn't the bug itself, it's how boring the fix was next to how big the story is. Heartbleed at least had a scary logo and a bleeding-heart graphic. This is just an old shell parsing function definitions in a way nobody thought to question for two and a half decades, and the fix is a package upgrade you can run before your coffee's done brewing. The hard part was just remembering everything you own that has a shell in it, which, if you've been doing this long enough, is more machines than you'd like to admit out loud.

Anyway. Patch your bash. Check your CGI. Talk to your in-laws about their WordPress installs. That was my Saturday.