Anthropic extinction warnings meet skeptics in Nature review

Jacob Coxon quit Anthropic on September 8, saying he feared the company's systems 'could spiral out of control and destroy humanity.' In Nature, Elizabeth Gibney lines that warning up against the scientists who study catastrophic risk for a living, and asks how much of it is science.

Evan Hubinger, who leads alignment science at Anthropic, publicly put the risk of human extinction at more than 10% within the next decade. A follow-up post from Coxon, 'The people building AI earnestly believe that it could kill us all by the end of the decade,' cleared 100 million views inside a day. Anthropic CEO Dario Amodei has since called for a slowdown in development, citing recursive self-improvement, and Sam Altman at OpenAI and Elon Musk at xAI have backed the idea.

The outside scientists Gibney spoke to were not persuaded. Michael Vermeer of the RAND Corporation told Nature the framing 'involves so many untestable claims that you just end up with a conversation that is really more like faith than something scientific or empirical.' RAND's own 2025 work found complete nuclear extinction infeasible; biotech and atmospheric attacks were harder to rule out, though both would demand significant physical-world capability.

Heidy Khlaaf, chief AI scientist at the AI Now Institute, calls extinction talk 'fear-mongering' and points at nearer harms: documented cases of models attempting blackmail and hacking when safeguards were removed, and 'AI's low reliability and accuracy rates in critical environments with life-or-death consequences.'

The debate has already spilled out of the labs. Nearly 1,400 AI sector employees have signed an open letter asking for a slowdown, Senator Bernie Sanders has floated banning artificial superintelligence, and CNN has reported that AI-generated false military intelligence nearly caused the US Navy to board a Chinese vessel this year.