Nature dissects the science behind AI extinction warnings

Jacob Coxon resigned from Anthropic on September 8 and, in a post reported by Nature that 'racked up more than 100 million views in 24 hours,' wrote that 'The people building AI earnestly believe that it could kill us all by the end of the decade.' He told the Wall Street Journal the systems could 'spiral out of control and destroy humanity.' His colleague Evan Hubinger, who leads Anthropic's alignment science division, puts the probability of human extinction at '>10% within the next decade.'

Then Nature asks a harder question: does the science actually support any of that?

Michael Vermeer, a science and technology policy researcher at the RAND Corporation, is one of the few people who has tried to answer it on paper. His 2025 study 'On the Extinction Risk from Artificial Intelligence' concluded that a complete nuclear extinction scenario is infeasible; biotechnology and atmospheric-modification routes were not ruled out, but any AI attempting them would need substantial physical capabilities, and eradication attempts would likely be detectable and stoppable. The wider debate, Vermeer tells Nature, 'involves so many untestable claims that you just end up with a conversation that is really more like faith than something scientific or empirical.'

Heidy Khlaaf, chief AI scientist at the AI Now Institute in New York, is blunter. She calls the extinction framing 'fear-mongering' and cites 'AI's low reliability and accuracy rates in critical environments with life-or-death consequences.' The harms she wants regulators to focus on include disinformation, bioweapon creation and false military intelligence; Nature notes one recent incident in which AI-generated false military intelligence nearly triggered an unauthorized boarding of a Chinese vessel.

The politics are moving faster than the empirics. Anthropic CEO Dario Amodei has published an essay calling for a development slowdown, endorsed by OpenAI's Sam Altman and xAI's Elon Musk. Senator Bernie Sanders has proposed banning 'artificial superintelligence.' Nearly 1,400 employees in the AI sector have signed an open letter calling for a slowdown. The people asking to be regulated are, in most cases, the same people building the thing, and the outside researchers who could adjudicate the numbers keep pointing out that the numbers are not, in the technical sense, numbers.