First post of the year, and it's not a predictions piece, because I hate predictions pieces. Every January every blog does the "here's what's coming in tech this year" post and half of them are wrong by March. Instead I want to tell you what I actually did over the break, which was drag my old ThinkPad T480 out of the closet and turn it into a little local AI box.
Backstory: I've had this laptop since 2019. It's got a Core i5-8350U, 16GB of RAM I upgraded myself with a $40 kit off Amazon a few years back, and a battery that lasts maybe 90 minutes now if you're generous with it. It's been sitting in a drawer doing nothing since I got my M2 MacBook Air. Every few months I feel guilty about that and think "I should do something with this," and then do nothing. This time I actually did something.
The something was Ollama. If you haven't touched it, it's a command-line tool (with a couple of GUI wrappers now) that lets you download and run open-weight language models locally instead of hitting an API. Install is genuinely one line on Mac and Linux, and there's a Windows installer too now. I put a fresh install of Ubuntu on the ThinkPad first (wiped the ancient Windows 10 install that was still limping along with about eleven browser toolbars I swear I never installed) and then ran the Ollama install script over SSH from my main machine.
The obvious problem: no GPU. This is integrated Intel graphics from 2018, so we're talking pure CPU inference. I wasn't expecting much. I pulled down a quantized 7B model first (the 4-bit version is maybe 4GB on disk) and honestly it runs. Not fast, you can watch the words show up one at a time like it's 2004 dial-up, roughly 3-4 tokens a second, but it runs, and for short questions that's tolerable. I would not want to write a novel through it.
A few things I'd tell past-me if I were doing this again:
- Get the quantized versions. The full-precision weights will just sit there swapping to disk and you'll wait five minutes for a "hello."
- Close everything else. This machine does not have headroom to spare, and if Firefox is open with your usual forty tabs it will choke.
- Set it up so you can hit it from your phone. I run it as a service on the ThinkPad and access it over my home network from a simple web UI (Open WebUI, which is free and took maybe ten minutes to get running in Docker). Now I've got a little private chat interface sitting on a machine under my desk that never leaves the house.
Is it as good as the big hosted models? No, obviously not, and I'm not going to pretend a quantized 7B model on a five-year-old ultrabook processor competes with anything running on a data center full of H100s. But here's the thing that actually surprised me: for maybe 60% of what I use a chatbot for day to day — rephrasing an awkward email, explaining a regex, brainstorming names for a side project — it's completely fine. I don't need frontier intelligence to ask "what's a better word for 'utilize' here." And there's something genuinely satisfying about knowing that whatever I type into it isn't going anywhere, isn't training anything, isn't sitting in some company's logs. It just stays on the box in my house.
The other honest reason I like this: it gave the old laptop a job again. I hate throwing out hardware that still works, and this thing still works fine, it just wasn't fast enough for modern web browsing with forty tabs open (see above, my own fault really). As a dedicated little inference box that sits closed most of the day and wakes up when I ping it, its age barely matters.
If you've got an old laptop or a spare desktop gathering dust, I'd genuinely recommend trying this over a weekend before you decide to recycle the thing. Total cost for me was $0 beyond electricity, since I already had the RAM upgrade from a couple years back. Worst case you learn Ollama's basically painless to set up. Best case your junk drawer laptop earns its keep for another few years.
Next thing on my list is trying a bigger model on it just to see how bad the slowdown gets, purely out of morbid curiosity. Will report back if it's interesting.