I finally sat down this week and did something I'd been putting off since roughly August: I went through my nginx logs on the box that runs this blog and actually counted who was hitting it.
Not "who" as in readers. Who as in bots. And it was worse than I expected.
I'd noticed the site felt a little sluggish loading old posts, the ones from like 2013 with fifteen tiny images in a gallery. My first guess was I'd finally hit some resource ceiling on the VPS, so I almost went and paid for a bigger plan. Glad I checked first, because that would've fixed nothing. When I grepped the access log for a single day, GPTBot alone made just over 40,000 requests. ClaudeBot was up around 19,000. Bytespider (that's TikTok's crawler, in case you haven't run into it) was doing its own thing at about 11,000, mostly hammering the same twelve posts over and over for no reason I can figure out. Amazonbot showed up too, politely, only a few hundred hits, so at least somebody's crawler has manners.
None of these are "attacks." Nobody's trying to break in. They're just scraping, presumably to feed into some model's training set or a RAG index somewhere, and they do not care even a little bit that my archive posts from 2012 have zero traffic value and are mostly me complaining about Windows Phone. robots.txt asks them nicely to slow down. Some of them read it. A lot of them don't, or read it and ignore the crawl-delay line because it's not a real standard, it's a suggestion that different bots interpret differently or not at all.
So here's what I actually did, in case anyone else running a small self-hosted blog wants the short version instead of reading six different forum threads like I did.
Step one: identify the worst offenders by User-Agent. This is just:
awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rn | head -30
Run that and you'll get an ugly but informative list. You'll recognize the browsers. Everything else is worth a second look.
Step two: block at the web server, not in robots.txt. robots.txt is a request. nginx config is a wall. I added a map block matching known bot user-agent strings and returning 403 for the ones I don't want, while leaving Googlebot and Bingbot alone because I still want to, you know, exist in search results. Something like:
map $http_user_agent $blocked_ua {
default 0;
~*GPTBot 1;
~*ClaudeBot 1;
~*Bytespider 1;
~*CCBot 1;
}
then in the server block, if ($blocked_ua) { return 403; }. People will tell you if in nginx is evil and to use a map-driven return instead for performance reasons, and they're not wrong exactly, but for a personal blog doing maybe 200 human visits a day, the performance difference is not something I am ever going to notice.
Step three: rate limit everyone else, bots included, that you haven't explicitly blocked. limit_req_zone keyed on IP, something generous like 10 requests a second with a burst allowance, so a crawler that respects rules but hits you fast still gets throttled instead of blocked outright.
I did not go the full Anubis proof-of-work route, the thing where visitors have to solve a little computational puzzle before they get in. I looked at it, it's clever, but it felt like overkill for a blog that's mostly me and about six RSS subscribers, one of whom I'm fairly sure is my brother.
The part that actually annoyed me, and this is the small complaint of the post: half the guides I found assumed you were running Cloudflare in front of everything, with their bot-fight-mode toggle just sitting there ready to flip. I don't run Cloudflare on this box. I like knowing exactly what's touching my server, and the VPS I've got through Tricknowtech is cheap enough and fast enough that I'd rather configure nginx myself than hand traffic control to somebody else's dashboard. Different tradeoff, not a knock on Cloudflare, just not my style for a hobby project.
After 48 hours with the blocks live, my daily request count dropped by something like 60%, and page loads on the old gallery posts are noticeably snappier. Bandwidth graph looks like it fell off a cliff, in a good way.
If you're running your own small site and haven't looked at your logs in a while, go look. You'll probably be annoyed too, and at least now you've got something to try.