It doesn't mean anything to control something smarter than you

[I feel guilty writing such an ~obvious blog post. But I still meet people who seem to be under the impression you can “control” something smarter than you.]

In worlds where we build superintelligent AI and things do not immediately go awry, we lose control;

For good or ill, the lightcone is no longer strictly “ours";

You are too distracted by the words “intelligence” or “control” as they're used at human-level if you do not understand this;

Take the “immediate shutdown” button.

You're in a room, staring at the internet, waiting for Claude Requiem 5.5 to make a suspicious move. Nearby is a red button, wired to plastic explosives, which snake around and through the server racks you're running Claude Requiem 5.5 on. Ok now what.

Under what conditions do you click the button? How easy would those conditions be to guess in advance for the AI? What about faking those conditions, for the AI? Technically achieving those conditions while the AI gets what it wants done? How much slack does Claude have, here?

A lot!

Because what you have here is not a system shaped like “humanity can shut the AI down whenever it wishes".

That's mistaking the purpose of the tool for the true shape of the tool.

Indeed the very concept of “tool” is somewhat misleading because the instant you built it, and back away, the tool might as well be any other part of the environment.

Perhaps more predictable-to-you than other parts of the environment—with more salient buttons you can press to get what you want—but it's not “you". The shutdown button is not us. It's not humanity.

The true shape of the tool is “specific human(s) Anthropic hired to sit in a room, presumably surveying a screen that leads to the internet [or downstream from others surveying the internet], able to click a button whenever something they find suspicious happens.” There are a ton of implicit details here like “these humans have families”; “get tired and have to take turns”; “use real laptops to survey suspicious behavior connected to actual Anthropic office wifi"; “have political axes to grind"; “can be fired and replaced by Dario Amodei"; “are probably already captured by a few memeplexes online, as terminally online people are wont to be”; “respond to incentives Anthropic designs: do you trust Anthropic to make it clear they should err on the side of caution? That shutting down the servers for a while at the cost of ~millions-to-billions is an acceptable price for multiple false positives?”; etc.

There's no glorious crowd of humans watching the AGI like a hawk to ensure it doesn't make any false moves. There’s no “control” in any meaningful sense. That word just… ceases to be relevant.

Learn to unsee:

See also: “The tale of the slow-mo CEO”, where one is technically “CEO” but also that ceases to mean anything when things move so fast and so competently that you cannot understand what's going on.

See also:

“Perception as a measure of intelligence, and the point of AGI is to be outmatched at intelligence: implies we'll be outmatched-by-AGI at the perception cat-and-mouse game”; “whether the AGI gives us glorious-Apollonian-future or kills us all or pulls a partial-value-takeover, the button will never be pressed"; in scenarios where your AGI is actually an AGI, the people in the room will not see what they consider to be button-pressing levels of behavior"; “thing-in-principle is not thing-in-reality”; “like all things, the shutdown-button is difficult to engineer and will fail in real life, unless you dedicate enough g to solving it, which the labs do not currently seem keen on allocating?”; “you do not understand just how many avenues of attack there are in this plan"; “reality has a surprising amount of manipulable detail"; “the AGI need not be ‘evil’ or ‘unaligned’ to simply not listen to you”; “you don't know what you need—the aligned AI does—would you blame it for spoonfeeding you? Isn't this what you wanted? But would you call it ‘control’?”; etc.