So this has been a fun week. My Friday afternoon plan was to push a small CSS fix for a client site, watch the webhook fire, watch Travis do its thing, go make dinner. Instead I got a 502 from GitHub, tried again, got a hung connection, tried a third time and gave up and just SSH'd into the box and pulled manually like its 2009.
Turns out I wasnt losing my mind. GitHub has been getting hammered by what they're calling the largest DDoS attack in the site's history, going back to March 26th and still not fully resolved as I'm typing this. Its not random either, which is the part that actually makes it interesting instead of just annoying. The traffic is apparently being aimed specifically at two projects: a mirror of greatfire.org and a mirror of the Chinese-language New York Times, both of which exist to help people route around the Great Firewall. GitHub itself hosts the content, sure, but the attack seems designed to make GitHub as a whole so miserable to use that they'll cave and pull those specific repos down.
Some of the security folks poking at the traffic (I saw a writeup from the Netresec guys making the rounds) think the requests are being injected through ad and analytics scripts served off a huge Chinese search engine, meaning ordinary people just browsing the regular internet in China are unknowingly being turned into part of the botnet every time a page with that script loads. If thats actually whats happening that's a pretty different and much scarier kind of DDoS than the usual rented-botnet spam version. Nobody consented to anything, their browser just quietly starts lobbing requests at github.com in the background. I dont know enough about network security to verify the mechanics myself, but it's the most plausible explanation floating around right now and GitHub hasnt exactly said otherwise.
What I do know is what it looks like from the outside: intermittent outages, slow clone times, webhooks that fire late or not at all, and a status page thats been more red than green for going on a week now. GitHub put out a statement basically saying this is the biggest attack they've ever weathered and they're not taking the content down, which, good for them honestly. I dont love that my client's deploy pipeline is an unwilling casualty of a fight between a code hosting company and (probably) a government, but I'd rather deal with a flaky webhook for a week than have GitHub fold because some ad network got weaponized.
It does make me twitchy about how much of my actual paying work runs through one company's infrastructure though. I still remember when everyone kept their code on SourceForge and that felt just as permanent right up until it very much wasnt (RIP, also please never open SourceForge again, the installer bundling thing they did later was inexcusable). GitHub feels different because of git itself — every clone is a full copy of history, so in theory nothing is actually lost if the lights go out for a few days. In practice though my CI, my issue tracker, my deploy hooks, even the little status badge on my README, all of that lives specifically at github.com and none of it fails over gracefully. I spent about forty minutes yesterday setting up a mirror push to Bitbucket just so I'd have somewhere else to point people if this drags on much longer. Probably overkill. Felt better than doing nothing.
Anyway if your build's been acting weird this week its almost certainly not you. Check the GitHub status page before you start bisecting your webpack config at midnight, ask me how I know. Going to go stare at my terminal and hope the next push actually lands on the first try for once.