So Anthropic put out a blog post yesterday about their updated Claude 3.5 Sonnet model, and buried in the middle of it was the part that actually made me sit up: a beta feature called "computer use." The pitch is that you can point the model at a screenshot of your actual desktop and it will move the mouse, click buttons, open apps, and type into text fields on its own, looking at the screen the same way a person would instead of going through some special API hook for every app. Not "generate me some code that does X." Actually reach in and do X.
I've been messing around with automation tools since I was a teenager, and my first reaction honestly wasn't "wow, the future is here," it was "oh no, here we go again." Remember AutoHotkey? Or QuickKeys back in the OS 9 days? I had an AutoHotkey script in like 2013 that would open my invoicing spreadsheet, tab to a specific cell, and paste in a date, and I thought I was hot stuff. It broke constantly. Any time a dialog box popped up half a second late, or a window wasn't in exactly the pixel position it expected, the whole thing would start clicking on nonsense. I once had it happily "confirm" a delete dialog in iPhoto because the window coordinates were off by about 40 pixels. Lost a folder of vacation photos. Never fully forgave myself, or the software.
So when I read that Claude can now look at a screenshot and decide where to click, my brain went straight to that. Except this time it's not a dumb coordinate script, it's a model actually parsing what's on the screen and reasoning about it, which in theory means it should be a lot more resilient to a window moving around or a button changing label. Anthropic's own numbers on this are honestly not that flattering yet, something like 14-15% on an evaluation called OSWorld that tests general desktop tasks (fill out a form, use a spreadsheet, that kind of thing), and they're pretty upfront that a person doing the same tasks would just crush it. They're framing it as early and experimental, not a "buy this and fire your assistant" product, and I appreciate that they said so instead of the usual breathless launch copy.
Still. Think about what it actually means for a piece of software to have general permission to move your cursor and click things. Every automation tool I've ever used, you write it once for one specific task and you babysit it the first few times it runs. This is different: it's improvising, in real time, based on what it thinks it's looking at. That's the part that's genuinely new, and also the part that should make anybody with a browser full of logged-in accounts a little nervous. A model that misreads a screen and clicks "confirm" on the wrong thing isn't a hypothetical, its basically guaranteed to happen sometimes, the only question is how often and how bad.
I'm not trying to be the old man yelling at the AI cloud here. I think this category of tool is obviously where things are headed, and there's something almost funny about the fact that the endpoint of a decade of chatbots is "okay but can it just use the computer like a person would." That's a genuinely different shape of problem than writing better sentences. It's also, weirdly, kind of an admission that most of our software still doesn't have great APIs for this stuff, so the easiest path is to just let the AI drive the same interface we do, mouse and keyboard and all.
Anyway I'm not signing up to give any model control of my actual desktop this week. Maybe in six months once other people have found the weird edge cases for me. Given my iPhoto incident I think I've earned the right to let somebody else go first on this one.
One smaller thing from the same week worth a mention: Amazon's new Kindle lineup, including the color one, started showing up in preorder emails, and I'm mildly tempted by the Colorsoft even though I know from experience I'll use it for about three weeks before going back to my beat-up Paperwhite. Some habits just aren't worth automating away.