Been a minute since I posted anything here, closing in on two weeks, which for this blog is basically a hiatus. Nothing dramatic happened, just a stretch where the day job ate everything and the laptop sat there judging me from the corner of the desk. Anyway, I'm back, and I actually have something worth typing about.
PrismML put out Bonsai 2 27B this week, and it's the kind of release that gets buried under bigger headlines but is way more interesting to me personally. It's a compressed version of Alibaba's Qwen3.8 27B, and the whole thing fits in 5.9 GB. Five point nine. That's a number I had to read twice. A 27-billion-parameter model that used to need a proper GPU rig now fits on the same drive as my photo backups.
I downloaded it the night it showed up (fine, I was procrastinating on something else) and ran it on my aging ThinkPad, the one with 16GB of RAM and a fan that sounds like it's filing a complaint every time I open more than six browser tabs. It loaded. It actually loaded, and responded to prompts at a speed that wasn't embarrassing. Not fast, not something I'd want to use for real work, but not the three-minutes-per-sentence disaster I expected either.
Here's my actual opinion on this, since I'm supposed to have one and not just hedge: compression like this matters way more than another frontier model announcement. Everyone's been writing about GPT-6 Astra and whatever Gemini is calling its newest flagship this month, and sure, those are impressive in a "look what a few hundred million dollars of compute buys you" kind of way. But Bonsai 2 is the kind of thing that actually changes what a regular person with a regular laptop can do today, not in some roadmap-shaped future. I'd rather have a smaller model I can run on hardware I already own than a bigger one I have to rent access to through an API that might change its pricing next quarter.
I did briefly consider whether it'd run better off my laptop entirely, spun up the idea of throwing it on a small VPS just to get it off my own hardware and see if the extra headroom helped, something I've done before for other side projects over on Tricknowtech when I didn't want my own machine tied up for a weekend. Decided against it for this one, mostly out of laziness, but the option's there if I change my mind later.
The quality trade-off is real, to be clear. I asked it a few things I'd normally throw at a bigger hosted model and the answers were noticeably shorter on nuance, especially on anything requiring multi-step reasoning. It's not a replacement for the big stuff. It's a replacement for "I need a model running locally right now and don't want to wait on a download measured in tens of gigabytes."
What I didn't expect was how much I enjoyed just having it sitting there, offline, not phoning home to anyone. There's something almost nostalgic about running something entirely on your own machine again, no API key, no usage dashboard, no little banner telling you you're approaching a rate limit. I know that's not a universal opinion, plenty of people would rather have the bigger hosted model and never think about VRAM again, but for my own tinkering I'll take the smaller thing I actually own.
Also, completely unrelated tangent: I keep seeing headlines about zero-click ad-delivered spyware getting used against regular people, and every time I read one I think about how much of my own browsing happens through ad-funded sites without a second thought. Not solving that today. Just noting it bugs me more than it probably should, given I don't even run an ad blocker on half my devices out of sheer laziness.
Anyway. Bonsai 2 is on my drive, my ThinkPad fan has mostly recovered, and I'll probably write more once I've actually used this thing for something real instead of just poking at it for an evening.