I've been putting this off for like eight months, mostly because I assumed my desktop was too old and too weak to run any of this stuff without turning into a space heater. Turns out I was wrong, and also turns out I'm an idiot for waiting this long, because the whole thing took maybe ninety minutes start to finish once I actually sat down and did it.
Quick context for anyone new here (hi, welcome, this blog has been limping along since 2011 so you're in good company if you just found it): I've got a 2019-era desktop, Ryzen 7 3700X, 32GB RAM, and an RTX 2070 that I bought used off a guy in a parking lot for less than it was worth because he was "upgrading and needed cash fast." No idea if that story was true. Card's been fine. Anyway, 8GB of VRAM is not a lot by 2026 standards, and I figured that ruled me out of the whole local-model thing entirely.
It doesn't. Here's what actually worked.
What I installed
I went with Ollama because everyone keeps recommending it and I didn't feel like fighting with Python environments at 11pm on a Tuesday, which is when I do basically all my hobby computer stuff now that I have a toddler who wakes up if I so much as sneeze before 9am. Install was one command on Linux, one download on the exe side if you're on Windows. No drama.
For the model itself I didn't go for anything huge. I grabbed a quantized 8B model (Q4 quant, so it's small enough to actually fit in that measly 8GB of VRAM with room to spare) instead of trying to cram in something bigger and watching it crawl or just refuse to load. This is the part people get wrong constantly — they read a benchmark chart for the full-size 70B version of a model, get excited, download it, and then wonder why their machine sounds like a jet engine and takes four minutes to answer "what's 2+2." Match the quant to your hardware. It's not complicated but it's the step everyone skips.
The actual setup
- Install Ollama.
- Pull a small quantized model. I'd tell you to start with an 8B or smaller if your VRAM is under 12GB.
- Run it once from the terminal just to confirm it loads and responds without erroring out.
- If you want a chat window instead of staring at a terminal prompt, grab one of the free front-ends that talks to Ollama's local API. I use one that runs in the browser, points at localhost, and looks basically identical to the hosted chat interfaces everyone's used to by now.
- Turn your wifi off and try it. This is the fun part. It still works. Nothing left your house.
That last step is honestly the whole reason I wanted to do this. Not because I think the cloud AI companies are reading my grocery lists and plotting something sinister, I don't actually believe that, but because I like knowing I have a fallback that doesn't depend on somebody else's servers being up, or somebody else's pricing page staying the same, or my internet not going out (which it does, constantly, our provider is genuinely terrible and I will die on this hill).
Where it falls short
I'm not going to pretend this replaces the big hosted models for anything that requires real reasoning depth or long context. It doesn't. Ask it something that needs it to hold a lot of threads together and it starts dropping stitches pretty fast, and coding help beyond fairly simple functions gets shaky. What it's actually good for, in my experience over the last couple weeks of poking at it during lunch breaks, is quick text stuff — rewriting an awkward paragraph, summarizing a long email I don't want to read, brainstorming names for things. Boring, useful, unglamorous work. Which honestly describes most of what I actually need day to day anyway, so I'm not mad about it.
Power draw is the other thing nobody warns you about. My whole desktop pulls something like 250-300 watts under load when the model's actively generating, which isn't nothing if you're running it constantly. I'm not, so it doesn't matter much for me, but if you're thinking about leaving something like this running 24/7 as a little home assistant, do the electricity math first. I didn't, initially, and then I looked at my utility bill and had a small moment of clarity.
Anyway. If you've got a GPU sitting around from a few years back and you've been assuming it's too old for any of this, it probably isn't. Give it the ninety minutes.