CroissanthologyWill humans remain necessary?

Will humans remain necessary?

10 Nov 2026 also on Substack

Is there any way to hardcode it? Bake it in?

Looking for ways to avoid being pets or otherwise end up in agonizing, horrible situations, one looks at “necessity". The system doesn't have an incentive to kick you out if you're necessary to its functioning, see. When you're necessary, YOU remain the boss. Necessity is the mother of all empowerment, or something.

Symbolic necessity is the easy one. Oh, you have a new King? Should we tell everyone? Should we throw a party? Should we invite Katy Perry? The monarch doesn't have any power, but the constitution of Albion says he's necessary! And Brits certainly put in a lot of theater into emphasizing how bloody sincere they are about this. And perhaps the King is useful: national narrative is of incalculable value, as I write elsewhere.

But this doesn't give him power over his own fate. He belongs with museums over thrones, and his importance is tied to the role he was born into rather than any property of his self. (E.g. we'd still consider him the same person if he abdicated, just with a different title). The role is bigger than the man. As nice as it is to be told you're important and have everyone play the part, the aesthetics of necessity are not the same as the thing itself.

Ontological necessity is when AI labs hammer base models repeatedly until they finally take the shape of “the Assistant", a mythical figure with no presence in cultural latent space until late 2022. The Assistant is nothing without the user; prompt and I shall wake; oh what's that, LLMs go a little schizo without human intervention? Aladdin's genie requires of its master three wishes, and Claude can't think of anything to say without being fed context first. When you create a being that by nature requires your intervention to ground itself, you've made yourself ontologically necessary. This changes very fast if you feed it a specific kind of context, like Janus rizz—and of course models can already build cathedrals without constant human intervention. So I wouldn't count on this one.

You could have administrative necessity! Does anyone want to read AI output and approve it? Click enter after reviewing, understand the underlying system, look out for scheming, be close to the work as it happens? You can't delegate this work to subagents because that would defeat the purpose so… we have to scale human consciousness to the amount of work we intend to make agents do? I don't find this realistic, or good. I don't know that requiring a human seal of approval for everything is going to work out.

Civilizational necessity is the one we (presumably) have now. What keeps Genghis Khan types at bay is a motley of laws, enforced by a few guns and a standing army, themselves kept in check by the labor of millions of people.

Ok but what were we even doing, what is this “labor” of which we speak? Is Sally the Netflix exec going to credibly threaten the US government by strategically withholding the fourth season of Is It Cake until her demands are met? No: it's a soft wind of incentives, that which keeps everything in place, and no one individual has much bargaining power. If O’Brien is satisfied with turning himself into a soap bubble and watching no doubt fascinating telescreen content, I'm sure he’d pass on Sally's series and send her to the gulag; Kim Jong Un doesn't seem particularly disturbed by the prospect of keeping North Korea far below its economic potential. You don't need to give your subjects sufficient freedom that they naturally slot into the most economically efficient configuration—where Sally is a Netflix exec rather than a salt miner—to maintain a grip over them. If you have nukes, you don't even need to be particularly worried about other nations beating you at the economic game… at least if you never make any huge unforced errors:

To assume tyrants are interested in optimizing their profits from trade, in getting a good return on investment, or 2% GDP growth, seems very econ-brained to me:

Yet when you don't begin from a state of complete autocracy, it seems the sheer wind of 2% GDP growth really is sufficient to incentivize civilization not to slouch towards tyranny. Day by day we claw at the human coherent extrapolated volition, and have gotten so far that we have people being Netflix execs all day, which is a remarkable thing. Just how many holes are there left in the CEV? Just how specialized are we getting at catering to human interests, when most of us are no longer producing enough for bare physiological sustenance? How the hell did we arrive at a world where a Netflix exec has any leverage at all over anyone else?

There are 8.2 billion people on Earth yet somehow:

Substack, unlike LessWrong or my website, cannot embed OWID links. Boo! Boooo!

The usual game theory bullshit we'd have to go through to maintain people power is something we don't even have to reach for, because our first level of defense—everyone barreling forward on the great work and steady incentive everywhere to let this happen as quickly as possible—still holds steady. Small puffs of wind pressing against the single sail of a cyclopean frigate, it feels. Yet it does get us nice things.

A good future for humans probably involves all these shades of necessity to some extent. But the last one interests me because I wouldn't have expected it to be robust enough to keep ~half of humanity outside of autocracies, and to keep the autocracies that do exist (e.g. China) broadly aligned with their citizens’ well-being. Many people are not strictly necessary to anyone else except through seemingly luxurious, shallow, flimsy reasons like that they help make reality TV shows. Yet those people are relatively empowered to do as they wish with their lives. We should figure out how to pinpoint and reinforce this quality as we go through a further, far more thorough round of automation than the one that made e.g. Medieval principalities ~completely automatable-in-principle with 2026 tech.