I've been meaning to write this one up for a while because about four people have asked me the same question over the last couple months: "how do you actually run a local model that's worth using without turning your laptop into a space heater." Short answer: it's doable now in a way it just wasn't two years ago. Long answer is this post.
Backstory, briefly: I take the train into the city twice a week and the wifi on it is genuinely useless, like dial-up-in-2003 useless. I got tired of losing an hour of productive coding time to a spinner, so I finally sat down and set up a local coding assistant that doesn't need a connection at all. Here's what I landed on, and where it falls short, because it does fall short in places and I'm not going to pretend otherwise.
The setup
I'm running Ollama on a MacBook Pro with 32GB of RAM (M-series chip, a couple years old at this point, nothing fancy), pulling a quantized coding model in the 14B-ish range. That size is the sweet spot for me: small enough to load in under a minute, big enough that it's not just producing garbage. Anything under about 7B and I found myself fighting the output more than writing code myself, which defeats the purpose.
On the editor side I'm using the Continue extension in VS Code, pointed at the local Ollama endpoint instead of a cloud API. Setup is maybe fifteen minutes if you already have Ollama installed: pull the model, drop a config block into Continue's settings pointing apiBase at http://localhost:11434, restart the extension, done. The whole thing runs at maybe 8-10 tokens a second on my machine, which sounds slow written out like that but in practice is fine for autocomplete-style suggestions and short function-level chat. It is not fine for "rewrite this 400-line file," and I stopped trying to make it do that.
What it's actually good for
Small, contained stuff. Writing a regex. Explaining what a chunk of unfamiliar code does. Drafting a test for a function I just wrote. Renaming variables consistently across a file. Boring, bounded tasks where being 90% right and locally available beats being 99% right and stuck behind a train tunnel.
What it is not good for is anything that needs broad context across a big codebase, or anything where I actually want the best possible answer and don't mind waiting three seconds for it. For that I still reach for a hosted model over wifi when I have it. I don't think local models have caught up on raw capability and I'm skeptical they fully will anytime soon. The gap between what fits on a laptop and what runs on a rack of GPUs somewhere is still wide, quantization tricks aside. Anyone telling you local models have "basically caught up" is selling something, usually a course.
The annoying parts
Battery life tanks hard when the model's actually generating: I've watched my battery percentage drop like I'm running a video export. Fan noise too, though that's more of a running joke in my house at this point than a real complaint. And you have to actually remember to pull model updates yourself since nothing nags you about it the way a cloud app does. I went about six weeks running a stale build before I noticed there was a meaningfully better quant available.
One thing I didn't expect: I like not having my code snippets going anywhere, even boring code. It's not a huge privacy crusade or anything, I just like the quiet certainty that when I'm poking at something half-finished and ugly, nobody's servers are seeing it. That alone might keep me running the local setup even once I've got a decent connection again.
If you want to try it
Get Ollama, grab a model in the 7-14B range depending on your RAM (rule of thumb, you want roughly double the parameter count in GB free), wire up Continue or whatever your editor's equivalent is, and just use it for a week on your commute or wherever your connection is worst. You'll know within a day or two whether it's solving a real problem for you or just being a novelty. For me it turned out to be the former, mostly because the problem (dead train wifi twice a week) was real and specific enough that "good enough and always there" beat "great but sometimes unavailable."
Next thing on my list is trying to get one of these running on an old Linux box I've got doing nothing in a closet, just to see if I can offload it off my laptop entirely. If that works I'll write it up too.