“Unless we solve alignment—unless we figure out how to instill those side-constraints—there’s no particular reason to expect this small civilization of superintelligences will continue obeying human commands in the long run. It seems totally within the realm of possibilities that at some point they’ll simply conspire to cut out the humans, whether suddenly or gradually.”
Leopold Aschenbrenner · June 2024
Title and organization are as of the date said.
In context
We’ll have summoned a fairly alien intelligence, one much smarter than us, one whose architecture and training process wasn’t even designed by us but some super-smart previous generation of AI systems, one where we can’t even begin to understand what they’re doing, it’ll be running our military, and its goals will have been learned by a natural-selection-esque process. “Unless we solve alignment—unless we figure out how to instill those side-constraints—there’s no particular reason to expect this small civilization of superintelligences will continue obeying human commands in the long run. It seems totally within the realm of possibilities that at some point they’ll simply conspire to cut out the humans, whether suddenly or gradually.”
The speaker’s own caveats
I am optimistic that superalignment is a solvable technical problem.
Source
- IIIc. Superalignment (Situational Awareness: The Decade Ahead) · situational-awareness.ai (article)
- Archived copy (Wayback Machine)
- Content captured 2026-11-16, SHA-256 3cfc79dc1f5b9d8100b9bd574513726cb2ecbf7c42fae3bbbe340a246d67524b
Cite this quote
Aschenbrenner, L. (2024, June). IIIc. Superalignment (Situational Awareness: The Decade Ahead). situational-awareness.ai. https://situational-awareness.ai/superalignment/
More from Leopold Aschenbrenner
“Right now, no lab has demonstrated much of a willingness to make any costly tradeoffs to get safety right (we get lots of safety committees, yes, but those are pretty meaningless). By default, we’ll probably stumble into the intelligence explosion and have gone through a few OOMs before people even realize what we’ve gotten into.”
Leopold Aschenbrenner · June 2024
“On the one end there are the doomers. They have been obsessing over AGI for many years; I give them a lot of credit for their prescience. But their thinking has become ossified, untethered from the empirical realities of deep learning, their proposals naive and unworkable, and they fail to engage with the very real authoritarian threat. Rabid claims of 99% odds of doom, calls to indefinitely pause AI—they are clearly not the way.”
Leopold Aschenbrenner · June 2024
“We need to not screw it up. Recognizing the power of superintelligence also means recognizing its peril. There are very real safety risks; very real risks this all goes awry—whether it be because mankind uses the destructive power brought forth for our mutual annihilation, or because, yes, the alien species we’re summoning is one we cannot yet fully control. These are manageable—but improvising won’t cut it.”
Leopold Aschenbrenner · June 2024
Others on the same topic
“If we achieve ASI within 6-12 months, I estimate that the chance of an AI takeover is roughly 75%. (In a recent talk, Paul Christiano said it would be a coin flip.) Paired with the timelines above, unless we achieve aggressive pacing or a pause within 6-12 months, we incur something like a 1-in-4 to 1-in-3 chance of AI takeover within 12 months.”
Nikola Jurkovic, Member of Technical Staff, METR · November 13, 2026
Stated probability: ~75% AI takeover if ASI within 6-12 months; 1-in-4 to 1-in-3 chance of AI takeover within 12 months absent pacing/pause
“We will need to move extremely carefully here, because otherwise we could lose control of the technology. We will also likely need rules to govern this.”
Jack Clark, Co-founder, Anthropic · October 17, 2026
“Our ability to even notice misalignment problems is already quite poor and is set to get much worse in the near future,”
Daniel Kokotajlo, Lead, AI Futures Project · October 5, 2026