Techpad
Hunting Down Fifteen Years of Dead Links

Hunting Down Fifteen Years of Dead Links

· 4 min read
Tutorials bash blogging link rot site-maintenance web-archiving

Hunting Down Fifteen Years of Dead Links

So I finally did the thing Ive been putting off since roughly 2019: I sat down on a Saturday and ran a link checker against every single post on this blog, going all the way back to November 2011. Almost 900 posts. I did not expect to enjoy this. I did not, in fact, enjoy most of it. But I learned some things, and I fixed some things, and I want to write about it because Im guessing at least a few of you have a blog or a personal site old enough to have the same rot problem and havent looked either.

The script itself is dumb simple. I pulled every outbound href out of the markdown source with a quick regex, deduped them, and threw them at a loop that does a HEAD request (falling back to GET when a server refuses HEAD, which more of them do now than I remembered) and logs anything that isnt a 200 or a redirect that lands somewhere sane. Maybe forty lines of bash and curl, nothing clever. Took about ninety minutes to run against 2,400-some unique URLs because I was rate-limiting myself to not look like a jerk to anyone's server.

The results: 31 percent dead. Not "moved," not "redirected," just gone — 404, DNS failure, or parked domain squatting on the URL trying to sell me the same domain back. Almost a third of everything Ive ever linked to on this website no longer exists in the place I put it. Some of that is obvious in hindsight (RIP every link I ever made to a startups /blog directory before that startup got acquired and the acquirer nuked the subdomain), but some of it genuinely surprised me. A 2013 post I wrote about RSS readers linked to four different services. Two are dead outright. One redirects to a completely unrelated crypto exchange now, which is its own kind of funny. Only one of the four still resolves to anything like what it originally was.

The categories of dead link, roughly in order of how much they annoyed me:

  • Company blogs that got "consolidated." Someone in marketing decided the engineering blog and the product blog should merge into one CMS and every old permalink just evaporated instead of getting a redirect. This is the single biggest category by far, easily half my broken links.
  • Personal sites that expired. People stop paying for hosting, or they die, or they just move on. Nothing to be mad about here, just sad in a small way.
  • News sites that paywalled or restructured. A few links still technically resolve but now sit behind a subscription wall that didnt exist when I linked to them.
  • Shortened URLs from services that shut down. I used bit.ly for maybe eighteen months around 2012 before I got paranoid about exactly this problem, and sure enough a handful of those links are just gone now because whatever they pointed to also died, so the redirect has nowhere to redirect to.

What I actually did about it: for anything with a live Wayback Machine snapshot, I swapped the link to point at the archive.org copy instead of quietly deleting the reference. I think thats the right call, even though it means clicking through my old posts now sometimes gets you the Internet Archives banner instead of the real site. Better than a 404. For the truly gone stuff with no snapshot anywhere, I mostly just left it, honestly, with a little (dead link) note. Rewriting history by deleting the reference felt worse than leaving an honest little gravestone.

One thing I will complain about, because this is my blog and I get to: it is wild to me how casually big companies treat their own URL history. A permalink is supposed to be permanent, thats the entire point of the word, and yet every couple years some team redoes a CMS migration and just lets a decade of inbound links from other sites rot without so much as a redirect map. It costs basically nothing to keep a redirects.json file around. I dont think Im being naive here, Ive built exactly that file for this blog twice now across platform migrations, and it took an afternoon each time.

If youre running an old site and havent done this audit, Id genuinely recommend it, if only for the nostalgia hit of rereading your own thirteen-year-old opinions about which RSS reader was going to win. Spoiler: none of them won. We all just ended up back on email newsletters, which is its own strange loop I dont have the energy to get into today.