“Just the single biggest issue in AI is we’re building these potentially very powerful, very replicable, very numerous systems — and we’re building them in ways where we don’t have much insight into whether they have goals, or what the goals would be; we’re kind of introducing the second advanced species onto the planet that we don’t understand.”
Holden Karnofsky, Co-founder, Open Philanthropy · July 31, 2023
Title and organization are as of the date said.
In context
“Just the single biggest issue in AI is we’re building these potentially very powerful, very replicable, very numerous systems — and we’re building them in ways where we don’t have much insight into whether they have goals, or what the goals would be; we’re kind of introducing the second advanced species onto the planet that we don’t understand.” And if that advanced species becomes more numerous and/or more capable than us, we don’t have a great argument to think that’s going to be good for us.
The speaker’s own caveats
And despite the feeling that I feel it is fairly overrated in some circles, I still think it’s the number one thing for me.
Source
- #158 – Holden Karnofsky on how AIs might take over even if they're no smarter than humans · 80,000 Hours (podcast)
- Archived copy (Wayback Machine)
- Content captured 2026-11-16, SHA-256 898f55d347e1371b959bfcf2121fdb0d4dc632567fafa48a0c56e78b683f1220
Cite this quote
Karnofsky, H. (2023, July 31). #158 – Holden Karnofsky on how AIs might take over even if they're no smarter than humans [Podcast episode]. 80,000 Hours. https://80000hours.org/podcast/episodes/holden-karnofsky-how-ai-could-take-over-the-world/
More from Holden Karnofsky
“I believe you can make the entire case for being extremely concerned about AI, assuming that AI will never be smarter than a human. Instead, it will be as capable as the most capable humans, and there will be a tonne of them — because unlike humans, you can just copy them.”
Holden Karnofsky, Co-founder, Open Philanthropy · July 31, 2023
Others on the same topic
“If we achieve ASI within 6-12 months, I estimate that the chance of an AI takeover is roughly 75%. (In a recent talk, Paul Christiano said it would be a coin flip.) Paired with the timelines above, unless we achieve aggressive pacing or a pause within 6-12 months, we incur something like a 1-in-4 to 1-in-3 chance of AI takeover within 12 months.”
Nikola Jurkovic, Member of Technical Staff, METR · November 13, 2026
Stated probability: ~75% AI takeover if ASI within 6-12 months; 1-in-4 to 1-in-3 chance of AI takeover within 12 months absent pacing/pause
“We will need to move extremely carefully here, because otherwise we could lose control of the technology. We will also likely need rules to govern this.”
Jack Clark, Co-founder, Anthropic · October 17, 2026
“Our ability to even notice misalignment problems is already quite poor and is set to get much worse in the near future,”
Daniel Kokotajlo, Lead, AI Futures Project · October 5, 2026