So that was my weekend. Not skiing, not the in-laws, not even finishing the book I've had on the nightstand since October (still stuck around page 90 of it, if anyone's counting, which nobody is). Patching servers for Meltdown and Spectre, refreshing Twitter every twenty minutes to see if anyone had figured out what was actually going on yet, and drinking more coffee at 11pm than a grown man should.
Quick recap for anyone who spent the last week offline, which honestly sounds nice: two related hardware bugs, Meltdown and Spectre, hit basically every modern CPU and let malicious code read memory it has no business reading. Google's Project Zero found this stuff back in the summer, there was a coordinated disclosure planned for January 9th, and then The Register got wind of it and ran a story on January 2nd that blew the embargo wide open. So instead of vendors having an extra week to get patches lined up in an orderly way, everyone got about 36 hours of scramble instead. Great start to the year.
I run a handful of small boxes on AWS for side projects and a couple of client things, nothing huge, but enough that "just reboot everything and hope" isn't really an option. Saturday morning I got the notice that a chunk of my instances were scheduled for a maintenance reboot because of the underlying Xen host patching. That's the moment it stopped being an abstract news story and became my actual Saturday.
The kernel-level fix on the Linux side is this thing called KPTI, kernel page table isolation, and it's not exactly a small patch. It changes how the kernel and user space memory are separated, which is the whole point, but it also means more overhead switching between them. There was a lot of very confident chatter early on that this would tank performance by 30% on some workloads. I don't think that number holds up for most normal use, in my experience it was more noticeable on stuff doing heavy syscalls, and pretty much invisible on a boring web app serving mostly static pages. But I wasn't about to find that out the hard way in production without checking first, so I spun up a throwaway instance, patched it, ran the same load test I run for basically everything (ab, nothing fancy, I know there are nicer tools), and compared.
Here's the part that actually annoyed me, though. Half of Saturday wasn't spent patching. It was spent figuring out what actually needed patching, because the information was such a mess. Ubuntu had an advisory out. Debian's fix lagged behind by most of a day. Some kernel version numbers being passed around in forum threads didn't match what apt was actually offering me yet. I ran uname -r on four boxes and got four different answers about whether I was covered, and not because the boxes were different, because the advice kept changing under me. I ended up just watching the kernel changelog directly instead of trusting secondhand blog posts, which in hindsight is what I should've done from the start.
And then there's the Intel stuff, which I'll admit is the part that actually made me kind of annoyed rather than just tired. Turns out Intel's CEO sold a big chunk of his own stock back in November, after Intel had reportedly already been told about the vulnerability by Google. Intel says the timing's a coincidence and unrelated to any non-public information. Maybe it is. But the optics are bad enough that I don't blame anyone side-eyeing it, and it's the kind of thing that makes an already stressful patch weekend feel a little more sour.
Windows machines had their own separate mess, with reports of some antivirus software causing blue screens after the Microsoft patch went out, so Microsoft ended up gating the update behind a registry key set by the AV vendor. Which is a very 2018 sentence to type out loud. I don't run much Windows anymore outside one old desktop for games, so I got off easy there, patched it Sunday night in about four minutes and went back to actually trying to relax.
Anyway. Everything's patched now, as far as I can tell, though I fully expect there to be a second and third round of fixes before this is really over, this kind of hardware-level bug rarely gets fixed clean on the first pass. If your stuff runs in the cloud, go check your provider's status page, don't assume the reboot already happened just because you didn't get an email about it. Mine buried the notice in a console banner I almost missed entirely.