So I was half paying attention to the Olympics feed yesterday (still annoyed about the tape delay thing, but thats a rant for another post) when I started seeing this story scroll by about a trading firm basically melting down in front of everyone. Knight Capital Group. If you havent heard of them, dont feel bad, I hadnt either until yesterday. Theyre one of those market-maker outfits that handles a huge chunk of retail stock trades routed through places like TD Ameritrade and Scottrade, the kind of company that only makes headlines when something goes catastrophically wrong.
Yesterday morning, something went catastrophically wrong.
Right at the opening bell, 9:30am Eastern, Knights trading software started doing something nobody asked it to do. It began flooding the market with erratic buy and sell orders across something like 148 different stocks, buying high and selling low over and over in a pattern that made zero financial sense. It kept doing this for 45 minutes before anyone managed to shut it off. By the time it stopped, the company had lost around $440 million. Not lost as in "the market was down." Lost as in their own software bought and sold its way through nearly half a billion dollars of bad trades before a human could pull the plug.
The stock cratered. Down something like a third in a single day, and it kept sliding after that. There's real talk now about whether Knight even survives this as an independent company.
Heres the part that got me, and the reason Im writing about this instead of just bookmarking the article and moving on: this looks like a deploy problem. From what's being reported, Knight had rolled out new trading software the previous week, and old code that was supposed to be dead, a leftover test function nobody had touched in years, got left active on one of their servers when the update went out. One server out of eight didnt get updated cleanly. That's it. That's the whole story. Not some exotic algorithm gone rogue, not a hacker, just a stale deployment and a flag that shouldnt have still existed turning back on at the worst possible moment.
I dont work in finance and Ive never touched a trading system in my life, but Ive shipped code to production plenty of times, and this is exactly the nightmare scenario every developer has had at 2am at least once. You push an update. Most of the fleet gets it. One box doesnt, for whatever dumb reason, a hung deploy script, a bad rsync, someone fat-fingering a hostname, and now you've got old code and new code running side by side pretending to be the same system. Ive had this happen with something as low-stakes as a WordPress plugin update leaving a stale cache file around. It just breaks a page layout for an hour, I notice, I fix it, nobody dies. Knights version of that same basic mistake cost them $10 million a minute.
What bugs me is how avoidable this category of failure is supposed to be by now. We have configuration management tools, we have canary deploys, we have the ability to roll a change out to one percent of servers and watch it before going wide. None of this is exotic in 2012. And yet here's a firm moving billions of dollars a day apparently pushing updates the same way a lot of us push updates to a personal blog: cross your fingers and hit deploy on all the boxes at once. Ill admit theres probably more nuance to their setup than one paragraph in a news story can capture, and Im sure their engineers are not amateurs. But the postmortem, whenever it actually comes out, is going to read like a horror story to anyone whos ever managed a fleet of servers.
Theres also something almost funny, in a bleak way, about a company whose entire business is high-frequency trading getting taken down by code thats been dormant since roughly the era of dial-up. Nobody deleted it because deleting old code feels riskier than leaving it alone, right up until the day it isnt. Ive got functions in projects from two years ago I keep meaning to rip out for exactly this same reason and keep not doing it, because it works, dont touch it. I dont have $440 million riding on mine though.
Anyway. Watch this one. Given how fast the stock is dropping, I wouldnt be shocked if Knight needs a buyer or a bailout within the week just to keep the lights on.