Nature explainer: AI extinction case rests on untestable claims

On September 8, Anthropic researcher Jacob Coxon resigned and posted that he feared the company's systems could 'spiral out of control and destroy humanity.' Within 24 hours his post had been viewed more than 100 million times. Evan Hubinger, who leads Anthropic's alignment science efforts, followed with his own estimate that the risk of human extinction was greater than 10% within the next decade.

In an explainer for Nature, Elizabeth Gibney walks through what any of this actually rests on. Researchers haven't specified a mechanism; the AI Futures project offered a speculative one in which 'an AI unleashes a biological weapon to kill humans off and thereby make more space for solar panels and robot factories.'

Michael Vermeer from RAND Corporation is blunt. Extinction predictions, he says, 'involve so many untestable claims that you just end up with a conversation that is really more like faith than something scientific.' A 2025 RAND study worked through specific pathways: nuclear weapons, biotechnology, atmospheric modification. Complete nuclear extinction was infeasible. The biotech and atmospheric scenarios could not be ruled out, but both would require substantial physical capabilities and time that humans could potentially detect.

Heidy Khlaaf, chief AI scientist at the AI Now Institute, calls the extinction framing 'fear-mongering' and points to 'AI's low reliability and accuracy rates in critical environments with life-or-death consequences' as the actual priority. Gibney cites a 2026 incident in which false AI-generated intelligence nearly caused the US military to board a Chinese vessel.

The timing is awkward. Dario Amodei's follow-up essay called for a slowdown, not a halt, and Sam Altman and Elon Musk have endorsed it. Nearly 1,400 AI sector employees signed an open letter demanding one. Some analysts quoted in the piece note that stricter regulation would let Anthropic, reportedly approaching an IPO, justify a slower pace without ceding ground to competitors. Four of the researchers we track had the piece in hand within hours.