Last Wednesday, GitHub got knocked over by what's being called the biggest DDoS attack anyone's ever recorded — 1.35 terabits per second of garbage traffic, for about eight minutes, before Akamai's Prolexic service kicked in and started scrubbing it. Total outage time was something like ten minutes. GitHub.com and some of the API went flaky, people panicked on Twitter for a bit, and then it was over and everyone moved on with their day.
Except I can't stop thinking about the "how," because it's such a dumb, almost embarrassing mechanism once you look at it.
The attack used memcached servers as amplifiers. Memcached, if you haven't run into it, is a caching layer a lot of web apps use to keep frequently-hit data in memory instead of hammering the database every time. Totally normal, totally boring piece of infrastructure. The problem is that some huge number of memcached installs out there are sitting directly on the public internet with UDP support turned on, no auth, no nothing, because that's how it ships by default and nobody bothered to lock it down.
So here's the trick: you spoof your source IP to look like it's coming from your victim (GitHub, in this case), then you send a tiny request to one of these open memcached boxes. The box dutifully replies, but it replies to the spoofed address, not to you, and the reply can be tens of thousands of times bigger than the request. People are throwing around an amplification factor of something like 51,000x. So an attacker with maybe a gig or two of actual outbound bandwidth can point a firehose at someone else that's over a terabit wide. You don't even need a botnet for this one, which is what makes it so unsettling. A handful of misconfigured servers did more damage here than the Mirai botnet managed with hundreds of thousands of hijacked cameras and routers back in 2016.
And word's already going around (Arbor Networks flagged it) that somebody else, an unnamed US service provider, got hit with an even bigger one just a few days after GitHub's, something in the 1.7 Tbps range. So this wasn't a one-off stunt against a high-profile target, it's just what happens now that this technique is public. Cheap, easy, and apparently every network with an exposed memcached box on port 11211 is a free amplifier for anyone who wants one.
I went and checked my own boxes after reading about this, which I'll admit felt a little paranoid but also just seemed like basic hygiene. I don't run memcached anywhere on my current setup, thankfully, though I definitely did years ago on an early version of this very blog, running a self-rolled WordPress caching layer around 2013 that I'm now fairly sure I never firewalled properly. That server's long gone, but it's a good reminder that "it's just internal caching, who'd bother" is not a security posture, it's a wish. I did find a Redis instance on an old side project listening on its default port with no password, which is its own kind of dumb, so that was a fun ten minutes of my Saturday fixing that instead of doing literally anything else I'd planned.
What bugs me most is how avoidable this whole class of problem is. Memcached didn't need UDP support exposed to the internet for basically anyone running it. Most people are using it for exactly what I was, a local cache for a local app, no business being reachable from outside at all. Same story you see over and over with exposed Redis instances, open MongoDB installs with no auth (there was a wave of those getting ransomed last year), routers with UPnP just wide open because turning it off means one extra checkbox. Defaults matter enormously and nobody treats them like they do until something like this happens.
I keep my own hosting pretty boring these days on purpose — small VPS, minimal services actually listening, and for the domain and deploy side of things I just push to a git remote through Tricknowtech and don't spin up extra reverse proxies or caching layers I don't strictly need. Less surface area, fewer 2 a.m. Saturdays spent grepping through netstat output. GitHub survived this one fine, obviously, they've got Akamai on speed dial. The rest of us running smaller stuff should probably go check what ports we've got open right now, because chances are you've forgotten about at least one.