I spent most of Sunday night doing something I probably should've done months ago: getting a language model running locally instead of just poking at ChatGPT in a browser tab like everyone else on the internet. Meta put out Llama 3.2 a few weeks back, including a couple of genuinely small versions (1B and 3B parameters) built to run on phones and laptops instead of a server farm somewhere. I've been meaning to try one since I read about it, and this week I finally sat down and did it.
Short version: it works, it's kind of amazing, and my 2020 MacBook Air got warmer than I expected.
Getting it running
I'm not going to pretend this is hard, because it genuinely isn't anymore. I used Ollama, which has been around for a while now and has gotten stupidly easy to use. You install it, open a terminal, and type:
ollama run llama3.2
That pulls the 3B model by default (there's a :1b tag if you want the tinier one) and drops you straight into a chat prompt right there in the terminal. No API key, no signup, no usage limits, nothing leaving your machine. Takes maybe five minutes depending on your connection, most of which is just waiting on the download.
First thing I did was ask it to explain a regex I was too lazy to parse myself. It got it right, mostly. Second thing I did was ask it a trivia question about a movie from the 90s and it confidently made up a plot that never happened. So, standard small model behavior. Fast, private, and occasionally just lying to you with total confidence.
The laptop situation
Here's the part I didn't love. Even the 3B model had the whole aluminum case running noticeably warm within about ninety seconds of any real back and forth. I'm on a fanless Air, which I knew going in was a bad match for sustained CPU work, but I underestimated how bad. Anything longer than a couple of exchanges and I had to move it off my lap. Fine for late October. I would not want to be doing this in July.
That got me wondering whether it'd make more sense to run the thing somewhere else entirely and just SSH in. My blog itself sits on a cheap VPS through Tricknowtech, and for about a second I considered spinning up a bigger instance just to host a model and query it over the network instead of roasting my own laptop. Decided against it, mostly because the whole appeal of running this stuff locally is that it's local. The second you put it back on a server you're basically rebuilding the thing you were trying to get away from, minus the OpenAI bill. Might revisit that idea for something that actually needs real horsepower, like image generation. Not for this.
Is it actually useful
Depends what you're using it for. It's not going to replace GPT-4o or Claude for anything requiring real reasoning or long context, the 3B model has a context window and a brain to match, both on the small side. But for quick, throwaway stuff, rewriting an awkward sentence, summarizing a paragraph I pasted in, generating boilerplate, it's genuinely fine. And it's fine in a way that doesn't involve my data going anywhere. That's worth something to me even when the output is a notch below what I'm used to.
I don't think local models are about to replace cloud ones for anyone doing serious work. But the fact that a 3 billion parameter model can run completely offline on a five year old laptop and still be useful is kind of wild if you remember what "AI on a laptop" meant even two years ago. Back then it meant autocomplete. Now it means an actual conversation, even if that conversation occasionally invents a 90s movie plot with complete confidence.
If you've already got Ollama installed, ollama run llama3.2:1b is worth five minutes just to see how fast the tiny one is. It's almost instant, and for basic stuff it barely feels like a downgrade.