Meta dropped Llama 3.1 this week, on Tuesday, and the number everyone's fixated on is 405. As in 405 billion parameters, the biggest open-weights model anyone's released that's actually competitive with the closed stuff. Zuckerberg wrote a whole essay about it, "Open Source AI Is the Path Forward," which is a pretty big swing for a company memo, but fine, I'll bite.
I've been running smaller local models on this machine for about a year now, mostly Llama 3 8B and a couple of Mistral variants, just to poke at them and see what a "small" model can actually do without hitting an API. So naturally my first move Tuesday night was to see if I could pull the new 405B down and run it myself.
I could not.
The full-precision weights are something like 810GB. Even the fp8 version Meta shipped alongside it is over 200GB by itself. My NAS has room, technically, but "technically has room" and "can move 200+GB without falling over" are different claims, and my home connection made that very clear around 1am when the download was still crawling along and my wife asked why the wifi was slow enough that her show kept buffering. I paused it. Some things aren't worth the argument.
The bigger problem isn't storage anyway, it's that even if I had the full weights sitting on disk, actually running inference on a 405B model needs something like 8 high-end GPUs just to hold it in memory at reasonable precision. I have one GPU. It is not a data center GPU. This was never going to happen on my setup and I sort of knew that going in, I just wanted to see how far I'd get before reality caught up with me.
So instead I did what apparently a lot of people are doing this week, which is hitting it through one of the hosting partners instead. Groq's been the one everyone's talking about because of how fast their chips spit tokens out, and Together AI and Fireworks both had it up within a day of the release too. Running it that way costs real money per token but it's nothing compared to buying the hardware, and it's actually usable today instead of theoretically usable in six months once I've saved up for a GPU cluster I don't have space for.
For what it's worth, the model itself is good. Not GPT-4o-good in every single test I threw at it, but close enough on most of the writing and reasoning stuff I care about that the gap feels more like a rounding error than a category difference, and Meta's benchmark claims put it right up against Claude 3.5 Sonnet too, which came out back in June and has honestly been my daily driver since. Whether Llama 3.1 405B displaces that for me, I doubt it, mostly because Sonnet's fast and this thing, run through an API, has a noticeable pause before it starts talking. But having a genuinely frontier-tier model with open weights sitting out there for anyone to download (if they've got the hardware, which, again, I do not) feels like a real shift, not just a PR move.
Also, quick unrelated aside: the Olympics opening ceremony was on Friday night, the one on the Seine in the rain, and I had it on in the background the whole time I was fighting with this download. The whole thing ran something like four hours and by the end I genuinely couldn't tell you if the metal horse and the headless woman singing were meant to be unsettling or if that's just how it read to a tired guy babysitting a stalled 200GB transfer at midnight. Either way, weirder television than I expected from an opening ceremony, and a decent way to kill time while a progress bar barely moves.
Anyway. If you're thinking about trying the 405B model yourself: don't bother self-hosting unless you've genuinely got the rack space and the GPUs sitting around already, just go through Groq or Together and pay by the token. I wasted an evening finding that out the hard way so you don't have to.