“We need to not screw it up. Recognizing the power of superintelligence also means recognizing its peril. There are very real safety risks; very real risks this all goes awry—whether it be because mankind uses the destructive power brought forth for our mutual annihilation, or because, yes, the alien species we’re summoning is one we cannot yet fully control. These are manageable—but improvising won’t cut it.”
Leopold Aschenbrenner · June 2024
Title and organization are as of the date said.
In context
“We need to not screw it up. Recognizing the power of superintelligence also means recognizing its peril. There are very real safety risks; very real risks this all goes awry—whether it be because mankind uses the destructive power brought forth for our mutual annihilation, or because, yes, the alien species we’re summoning is one we cannot yet fully control. These are manageable—but improvising won’t cut it.” Navigating these perils will require good people bringing a level of seriousness to the table that has not yet been offered.
The speaker’s own caveats
These are manageable—but improvising won’t cut it.
Source
- V. Parting Thoughts (Situational Awareness: The Decade Ahead) · situational-awareness.ai (article)
- Archived copy (Wayback Machine)
- Content captured 2026-11-16, SHA-256 04bf0243bfe77a51d4a4dd380be38bfb98b28bbbfb1519780f7f1c8d6766ede1
Cite this quote
Aschenbrenner, L. (2024, June). V. Parting Thoughts (Situational Awareness: The Decade Ahead). situational-awareness.ai. https://situational-awareness.ai/parting-thoughts/
More from Leopold Aschenbrenner
“Right now, no lab has demonstrated much of a willingness to make any costly tradeoffs to get safety right (we get lots of safety committees, yes, but those are pretty meaningless). By default, we’ll probably stumble into the intelligence explosion and have gone through a few OOMs before people even realize what we’ve gotten into.”
Leopold Aschenbrenner · June 2024
“On the one end there are the doomers. They have been obsessing over AGI for many years; I give them a lot of credit for their prescience. But their thinking has become ossified, untethered from the empirical realities of deep learning, their proposals naive and unworkable, and they fail to engage with the very real authoritarian threat. Rabid claims of 99% odds of doom, calls to indefinitely pause AI—they are clearly not the way.”
Leopold Aschenbrenner · June 2024
“The AGI race has begun. We are building machines that can think and reason. By 2025/26, these machines will outpace many college graduates. By the end of the decade, they will be smarter than you or I; we will have superintelligence, in the true sense of the word. Along the way, national security forces not seen in half a century will be unleashed, and before long, The Project will be on. If we’re lucky, we’ll be in an all-out race with the CCP; if we’re unlucky, an all-out war.”
Leopold Aschenbrenner · June 2024
Others on the same topic
“On the current path, I think it is more likely than not that humanity loses control to these AIs, and it could end in human extinction,”
Jacob Coxon, Researcher, pretraining, Anthropic · October 5, 2026
“If the language models actually reach the capability threshold where they can shape the world unconstrained by human will, they will probably do something extreme and destroy humanity in the process.”
Daniel Selsam, Researcher, OpenAI · September 14, 2026
“I also work at an AI company and believe there’s plausibly a ≥10% chance that a future out-of-control AI kills everyone (IMO even 1% is unacceptably high). And higher still is the risk that we “only” get permanently disempowered by AIs that don’t deeply want the best for us.”
Dmitrii Krasheninnikov, Research fellow, AI safety (title from search snippet), Anthropic · September 11, 2026
Stated probability: ≥10%, “plausibly”; no time frame
All quotes on extinction risk →All quotes on loss of control →