So by now you've probably heard about Knight Capital blowing up on Wednesday morning. If you haven't: a market maker called Knight Capital Group flipped on new trading software right at the opening bell on August 1st, and within about 45 minutes it had flooded the New York Stock Exchange with a huge pile of erroneous orders across something like 140-some stocks. By the time somebody found the kill switch, the firm was sitting on losses being reported around $440 million. Forty five minutes. Thats less time than it takes me to get through my inbox most mornings.
Ive been reading everything I can find on this since Wednesday because its basically my nightmare scenario, just with a few more zeroes attached. The early reporting suggests the actual bug wasnt some exotic algorithm gone rogue, it was old code. A flag that used to trigger a test system years back apparently got repurposed for something else, and when the new release went out, not every server got updated cleanly, so at least one machine kept running against the old meaning of that flag. Multiply a tiny miscommunication like that by automated order routing at Wall Street volumes and you get an exchange getting hammered with garbage trades before any human even notices somethings wrong.
I dont work anywhere near Wall Street, obviously (my "production incidents" mostly involve a WordPress plugin and occasionally forgetting to close a MySQL connection on a cron job), but this is exactly the kind of failure that keeps me up at night with my own dumb little side projects too. Dead code sitting around because nobody wanted to be the one to delete it. A deploy that goes out to some but not all of the servers. No obvious kill switch when things start going sideways. Ive done every one of these things personally, just at a scale where the worst outcome is a broken page instead of nearly sinking a company that trades a meaningful chunk of US stock volume on a normal day.
The part that gets me isnt even the money, its that there apparently wasnt a fast way to just stop it. Youd think a firm moving that kind of volume would have a giant red button somewhere, obvious and well tested. Maybe they had one and it didnt work, I havent seen anything definitive on that part yet. But the lesson I keep circling back to is boring and completely unglamorous: the ability to turn a thing off fast is worth more than almost any feature you could ship. Feature flags, canary rollouts, staged deploys, whatever you want to call the practice, none of it is fun to write blog posts about, but this week is a pretty expensive advertisement for why it matters anyway.
Small tangent, but I cant get over how lopsided the timeline is here. 45 minutes of bad code running, versus what's probably going to be months of lawsuits, regulatory review, and Knight scrambling to find new capital just to stay solvent. Thats a wild trade between cause and consequence (pun sort of intended, sorry). One leftover flag nobody cleaned up, and suddenly a whole companys future is a question mark.
On a completely different note, Ive also been half-watching the news out of JPL this week, since the Curiosity rover is due to attempt its landing on Mars in just a couple more days, and everyone over there seems understandably nervous about the descent sequence people have been calling "seven minutes of terror." Its a nice contrast, honestly. One team has spent years testing every failure mode of a system that gets exactly one shot, no rollback, no reverting to the last known good deploy. The other apparently pushed something to a live trading system without anyone being fully sure it landed cleanly everywhere it needed to. Im rooting hard for the rover people this week, for what its worth.
If youre a developer reading this and you know you have some ancient conditional sitting in your codebase thats been dead for a year or two because deleting it felt scary, today might be a decent day to go find out what it actually does. Or just delete it and see what breaks. Thats basically a controlled version of what happened to Knight, minus the $440 million.