So I finally got Llama 2 running on my own machine last night, two days after Meta dropped it, and I have thoughts.
For anyone who missed it: Meta put out Llama 2 on Tuesday (the 18th), this time with Microsoft standing next to them at the announcement, which is its own small oddity given how much Microsoft has poured into OpenAI. The models come in three sizes (7B, 13B, and 70B parameters) and unlike the original LLaMA from back in February, these are actually licensed for commercial use, not just research. No begging for access through some Google Form that takes three weeks to process and then leaks onto BitTorrent within a day anyway (which, if you remember, is exactly what happened to LLaMA 1).
There's also a 34B model that Meta trained but didn't release. Their stated reason is that they didn't have enough time to "red team" it properly before launch. Make of that what you will: I find it a little funny that the size in between is the one they decided was too spicy to ship, while 70B, the biggest one, is apparently fine.
Anyway. I didn't bother with the official request form since I don't have a business use case and didn't feel like waiting on Meta's approval process. Within about a day of the release, TheBloke (if you spend any time on Hugging Face you know the name) already had quantized GGML versions of the 7B and 13B chat models up, so I grabbed those instead. Downloading a 4-and-a-half gig file on my home connection at 9pm on a Tuesday is not a fast experience, for the record. I started the download, made dinner, came back, it was still going.
Got it running through llama.cpp on the command line. The 7B chat model loads fast and responds fast, but the quality is kind of what you'd expect from a 7 billion parameter model: fine for short factual stuff, gets confused on anything with more than two steps of reasoning, and occasionally just restates the question back at me in a slightly different order like it's stalling for time. The 13B version is noticeably better, still not anywhere close to what GPT-3.5 does through the ChatGPT app, let alone GPT-4, but it's running entirely on my own hardware with zero API calls going anywhere, and that's genuinely the appeal here. No rate limits, no usage logs going to a server in Virginia, no $20 a month.
I did not attempt the 70B model. My desktop has 32GB of RAM and even the heavily quantized versions of that one want more than I've got, plus a GPU I don't own. Context window is up to 4096 tokens now instead of 2048, which sounds like a small bump written down like that but makes an actual difference once you're pasting in more than a paragraph or two of context.
One thing that bugged me: the chat formatting is oddly picky. There's a specific prompt template with system tags and instruction brackets that the model expects, and if you don't follow it closely the responses get noticeably worse: more rambling, less on-topic. Took me a couple tries with a template someone posted on a GitHub issue thread before I got outputs that felt coherent. Not exactly plug and play, and I'd guess most people who aren't already comfortable in a terminal are going to bounce off this entirely and just wait for someone to wrap it in a nicer app.
Also, unrelated, but I bought my Oppenheimer IMAX ticket for tomorrow night while my download was running, and the AMC app choked twice trying to process the payment. Apparently I was not the only one trying to lock in a seat before the weekend. Good use of forty-five minutes of my life, alternating between watching a progress bar and refreshing a seating chart.
Overall verdict on Llama 2 after one evening: it's not going to replace ChatGPT for me day to day, but as a thing to mess around with on my own terms, run offline, poke at, break, and not have to think about what OpenAI's usage policy says about it — that part I like a lot. I'll probably try the 13B model with some actual local documents this weekend and see how it holds up as something closer to a real tool instead of a toy.