Nature tests AI extinction warnings against the science

Nature has weighed the extinction scenarios circulating out of frontier AI labs against the empirical literature, where one researcher calls the underlying assumptions closer to faith than science. The review follows a September 8 resignation at Anthropic by researcher Jacob Coxon, who told the Wall Street Journal that the company's AI systems 'could spiral out of control and destroy humanity.'

Within 24 hours, Evan Hubinger, who leads Anthropic's alignment science division, reposted the warning with his own estimate that the risk of human extinction sits '>10% within the next decade.' The thread drew over 100 million views. CEO Dario Amodei followed with an essay calling for a slowdown, not a halt, in frontier development, and Sam Altman and Elon Musk endorsed the proposal. Roughly 1,400 AI-sector employees signed an open letter asking for development to slow after a wave of cybersecurity incidents.

Researchers outside the labs are unconvinced. The extinction case 'involves so many untestable claims that you just end up with a conversation that is really more like faith than something scientific or empirical,' said Michael Vermeer of the RAND Corporation. A 2025 RAND study examined nuclear weapons, biotechnology and atmospheric modification as mechanisms; it ruled out complete extinction by nuclear weapons alone and found other scenarios would require substantial physical interaction and would likely be detectable. Heidy Khlaaf of the AI Now Institute called the extinction framing 'fear-mongering,' arguing that disinformation, psychological harm, bioweapon creation and military miscalculation deserve more attention. Four researchers on our Who's Who list have shared the piece.

The commercial context is doing some of the talking. Stricter rules would validate Anthropic's slower cadence as it approaches an IPO, and David Sacks, co-chair of the US President's Council of Advisors on Science and Technology, has flagged 'massive product-liability exposure' if models enable severe cyberattacks.