Archiving your own internet

One of the recurring disappointments of my life is reading decade-old SSC posts, clicking on a link, and getting greeted by a 404. Linkrot is especially relevant when you're a prolific author with little regard for building carefully constructed libraries. SSC posts fade and fall apart: their “internet weight”—the linked context you can access easily—decays quickly. It feels like reading about the work of Celsus, a man known exclusively through a single work of criticism against him. For example the SSC post In Favor of Niceness, Community and Civilization is a quote-by-quote refutation of a lost Facebook rant by “Andrew Cord", itself sparsely backed up on a post that is now a 404.

SSC might not be a great example: Scott Alexander's audience is large and CS-literate, such that it's likely most outlinks got backed up at some point. But what of lesser-known authors of the time, who may yet experience a renaissance in history grad student dorms? Texts from antiquity are heavily selected for being relevant to Christianity, given medieval monk triage habits. SSC being an outlier among blogs in its time doesn't mean smaller blogs won't be prized as lost jewels in the future, depending on the future's new selection criteria. What of their internet weight?

Well, you build whatever monster automation Gwern built for himself on Gwern.net. Opus 4.7 made a comparison site for these two:

Scott lets his posts undergo an order of magnitude more rot! This follows from the two men's respective philosophies: Gwern wants a “long site”, and several of his posts took him years to write, like making a layered renaissance painting. Meanwhile Asterisk editor Clara Collier told me Scott's the only person she knows who requires zero editing after banging out a post and essentially never re-reads his work. Historically, SSC posts were written in between appointments at his hospital and were not his “main thing” like they were for Gwern. Gwern is humble with his explanations for himself, while Scott basically offers “I don't understand why people find it hard to write. They can talk right? Isn't writing just like talking?"

But this doesn't make SSC archiving less important for historians. So much of our literary corpus was never treated with respect by its own authors. You needn’t want to build a cathedral to be doing it anyway.¹

So systematic solutions to back up your outlinks and update them to backups are rare, but definitely a good idea for the health of our internet. You could figure out what Gwern and Achmiz did in their code base or you could use this code Claude made that lives in your macbook terminal and should be simple to set up:

https://github.com/Croissanthology/croissanthology.github.io/tree/main/archiver

The easiest way, as always, is just asking Claude in the CLI to do it! Anyway here's a little preview: