So Cloudflare dropped a blog post yesterday (that'd be August 4th) basically calling Perplexity a liar, and I have opinions about this because I've been staring at my own server logs for going on fourteen years now and the bot traffic situation has gotten genuinely unhinged.
Quick version if you missed it: Perplexity has a declared crawler that's supposed to respect robots.txt like every other well-behaved bot is supposed to. Cloudflare says that when site owners block that declared crawler, Perplexity just... keeps coming back. Different user agent, made to look like a regular Chrome browser on a Mac, routed through different networks entirely, hitting tens of thousands of domains. Not identifying itself. Not stopping when told to stop. Cloudflare's language wasn't subtle either — they used the word "stealth" right there in the post title, and when Cloudflare wants to pick a fight with you in public, that's not a company being coy.
Perplexity's response, predictably, was that this is overblown and their crawling is legitimate and user-initiated (the idea being that when you ask their assistant a question, it goes and fetches a page on your behalf, which is a different thing than indiscriminately scraping the whole internet). Maybe there's something to that distinction in some cases. But it doesn't really explain rotating ASNs and spoofed browser signatures to get around an explicit block. If you're doing something above-board you don't usually need three different disguises to keep doing it after someone's asked you to stop.
I'll be honest, I don't think this is really about Perplexity specifically being uniquely villainous. It's about what happens when every AI company on earth needs fresh web content to feed a model or answer a query, and the entire economic relationship the web has run on for twenty-five years (you let Google index you, Google sends you readers, everybody eats) has quietly stopped working. The crawlers still show up. The readers mostly don't anymore, because why click through when the chatbot already gave you the answer. That's the actual story under the story, and it's been building for a couple years now, this one Cloudflare post just put a specific, embarrassing name on it.
Which, small tangent, is why I went and checked techpad's own access logs last night instead of doing literally anything more productive with my evening. Turns out a pretty enormous chunk of my traffic every month is bots at this point, way more than it used to be even five years ago, and a meaningful slice of that doesn't announce itself as anything in particular in the user agent string. Could be research crawlers, could be some AI startup's scraper, could be nothing sinister at all. I genuinely can't tell from the logs alone anymore, and that bugs me more than it probably should for a blog that maybe forty humans read in a given week.
I don't have a robots.txt sophisticated enough to fight this and honestly I'm not going to build one. That's a losing arms race for a one-person blog even if it's a fair fight for Cloudflare, who literally sells "stop unwanted bots" as a product line and therefore has every incentive to make this sound as dramatic as possible. Worth keeping in mind when you read their version of events — they're not a neutral party here, they're a vendor whose business model benefits from you being scared of crawlers. Doesn't mean they're wrong about this one. It means read the primary source and not just the headline, which, yeah, I realize is a little rich coming from a guy currently writing a reaction post about a blog post.
Anyway. If you run any kind of site with content worth scraping, this is worth a read even if you don't care about Perplexity specifically, because the underlying mechanics (declared bots getting blocked, then coming back wearing a trench coat) are apparently not exclusive to one company. I'd bet money we hear a version of this story again before the year's out, just with a different name attached to it.