So Llama 2 is out. Meta and Microsoft announced it this morning tied to the Inspire keynote, and by the time I'd made a second cup of coffee my Twitter (I still can't make myself type the other name) was already full of people posting screenshots of chat completions. Three sizes this time: 7B, 13B, and 70B, plus chat-tuned versions of each. And the actual headline for me isn't the benchmarks, its that you can just have it. Fill out a form on Meta's site, give them an email, and a download link shows up. No BitTorrent magnet link, no waiting for someone's leaked archive to finish seeding at 40kb/s like back in March.
I filled the form out around 9:40am. The email landed at 9:52. Twelve minutes, which felt almost suspiciously fast for a company the size of Meta.
I want to be honest about something though: I did not download the 70B model. It's something like 130GB for the full-precision weights and I have neither the disk space nor the patience for that on a Tuesday. What I actually wanted was the 13B chat model running locally on my own machine, no API key, no rate limit, no wondering if my prompts are getting logged somewhere. That's the whole appeal of open weights for me. I don't necessarily think local models beat GPT-4 on anything (they don't, not close, not yet), I just like owning the thing I'm using.
Getting it running took longer than the download itself, mostly because I'm stubborn about not just using a hosted playground. Meta ships the weights in their own format, which meant converting to something llama.cpp can actually read. Georgi Gerganov's project has been unreasonably fast about supporting new model drops, and sure enough there was already a PR open for Llama 2 support before I'd finished the conversion script. By early afternoon someone going by TheBloke on Hugging Face had already uploaded GGML quantized versions of all three sizes, which saved me from doing the quantization myself. I grabbed the 4-bit quantized 13B, about 7.4GB, and it fit comfortably on my M1 MacBook with room to spare.
First prompt I threw at it was something dumb, "write a limerick about a printer that hates you," because that's genuinely the fastest way to tell if a model's chat tuning is any good or if it's going to answer like a customer service bot reading from a script. It did fine. Not amazing, a little stiff, but coherent and it actually rhymed, which some models still manage to mess up.
Here's my actual complaint, and it's a small one: the license. It's mostly open, free for research and commercial use, but there's a clause that says if your product has more than 700 million monthly active users you need to go get a separate license from Meta directly. Which, fine, that's obviously aimed squarely at Google and maybe Apple, and it's not going to affect a single person reading this blog. But it means "open source" isn't quite the right word for it and I wish people (including Meta's own PR) would stop using it that loosely. Permissively licensed, sure. Open source in the sense the OSI actually uses the term, not really.
Worth mentioning too, since it's been on my mind all week: Anthropic put Claude 2 out last week and it's still locked to the US and UK only, browser chat interface, no API access for most people who want it. So while half the internet is stuck refreshing a waitlist for a hosted product they can't even try yet, Meta just handed out actual model weights to anyone who filled in a form with a real-looking email address. Strange contrast for the same month. I don't think Llama 2 is a better model than Claude 2, from what little I've tested I doubt it's close. But I know which one I was actually able to use today.
Battery's at 11% and the fan on this laptop has been running since lunch. Going to go let it cool down.