“Safety progress takes time. If you don’t have enough time, at each stage of AI development, to ensure the AIs that you build in the next stage won’t destroy humanity, then you’ll fail, and humanity will be destroyed. Automated alignment research helps a lot here, and that’s a core reason I think it’s so important (human alignment researchers are too scarce and slow). But even if AIs are doing most or all of the alignment work, it still matters how much time they have.”
Joe Carlsmith · March 19, 2026
Title and organization are as of the date said.
In context
Introduction: the basic case for the importance of capability restraint is obvious. “Safety progress takes time. If you don’t have enough time, at each stage of AI development, to ensure the AIs that you build in the next stage won’t destroy humanity, then you’ll fail, and humanity will be destroyed. Automated alignment research helps a lot here, and that’s a core reason I think it’s so important (human alignment researchers are too scarce and slow). But even if AIs are doing most or all of the alignment work, it still matters how much time they have.”
Source
- On restraining AI development for the sake of safety · joecarlsmith.com (article)
- Archived copy (Wayback Machine)
- Content captured 2026-11-16, SHA-256 e2458e7bf6d41db5d1899a2a2c93bb9ed350961b872e2f35f677910cebb6dcb4
Cite this quote
Carlsmith, J. (2026, March 19). On restraining AI development for the sake of safety. joecarlsmith.com. https://joecarlsmith.com/2026/03/19/on-restraining-ai-development-for-the-sake-of-safety/
More from Joe Carlsmith
“Overall, my current view is that despite these uncertainties and trade-offs, we should try hard to put ourselves in a position to steer or restrain AI development when doing so is necessary for safety; that even limited or temporary success in this respect can make a significant positive difference; and that more thoroughgoing success remains, at least, a live possibility.”
Joe Carlsmith · March 19, 2026
Others on the same topic
“If we achieve ASI within 6-12 months, I estimate that the chance of an AI takeover is roughly 75%. (In a recent talk, Paul Christiano said it would be a coin flip.) Paired with the timelines above, unless we achieve aggressive pacing or a pause within 6-12 months, we incur something like a 1-in-4 to 1-in-3 chance of AI takeover within 12 months.”
Nikola Jurkovic, Member of Technical Staff, METR · November 13, 2026
Stated probability: ~75% AI takeover if ASI within 6-12 months; 1-in-4 to 1-in-3 chance of AI takeover within 12 months absent pacing/pause
“On the current path, I think it is more likely than not that humanity loses control to these AIs, and it could end in human extinction,”
Jacob Coxon, Researcher, pretraining, Anthropic · October 5, 2026
“If the language models actually reach the capability threshold where they can shape the world unconstrained by human will, they will probably do something extreme and destroy humanity in the process.”
Daniel Selsam, Researcher, OpenAI · September 14, 2026
All quotes on extinction risk →All quotes on loss of control →All quotes on pause →