On AI Risk
“Unless we solve alignment—unless we figure out how to instill those side-constraints—there’s no particular reason to expect this small civilization of superintelligences will continue obeying human commands in the long run. It seems totally within the realm of possibilities that at some point they’ll simply conspire to cut out the humans, whether suddenly or gradually.”

Leopold Aschenbrenner · June 2024

Title and organization are as of the date said.

Primary source verifiedExpertise: AI alignment, AI forecasting, economicsLoss of control

In context

We’ll have summoned a fairly alien intelligence, one much smarter than us, one whose architecture and training process wasn’t even designed by us but some super-smart previous generation of AI systems, one where we can’t even begin to understand what they’re doing, it’ll be running our military, and its goals will have been learned by a natural-selection-esque process. “Unless we solve alignment—unless we figure out how to instill those side-constraints—there’s no particular reason to expect this small civilization of superintelligences will continue obeying human commands in the long run. It seems totally within the realm of possibilities that at some point they’ll simply conspire to cut out the humans, whether suddenly or gradually.”

The speaker’s own caveats

I am optimistic that superalignment is a solvable technical problem.

Source

Cite this quote

Aschenbrenner, L. (2024, June). IIIc. Superalignment (Situational Awareness: The Decade Ahead). situational-awareness.ai. https://situational-awareness.ai/superalignment/

More from Leopold Aschenbrenner

All 5 quotes by Leopold Aschenbrenner →

Others on the same topic

All quotes on loss of control →

Report an error in this quote