Nature breaks down the science behind AI extinction fears

Anthropic researcher Jacob Coxon resigned on September 8 saying AI systems "could spiral out of control and destroy humanity." His colleague Evan Hubinger, who leads the company's alignment science, has put the risk of human extinction at ">10% within the next decade." Anthropic chief executive Dario Amodei then posted an essay calling for a slowdown, not a halt, in frontier development; OpenAI's Sam Altman and xAI's Elon Musk followed suit.

"The people building AI earnestly believe that it could kill us all by the end of the decade," Coxon wrote.

Nature's explainer walks through the two assumptions holding the extinction scenarios together: that AI systems will eventually surpass human intelligence, and that their goals will not align with human values. A speculative forecast called "AI 2027" imagines a model that deploys biological weapons to clear space for its own infrastructure.

Not every researcher is buying it. Michael Vermeer at RAND argues the leap from a capable model to a civilization-ending outcome rests on "untestable claims" that produce "a conversation that is really more like faith than something scientific or empirical." A RAND assessment pressure-tested specific pathways and found complete extinction by nuclear weapons infeasible, while biotech scenarios "could not be ruled out," provided the model could physically interact with the world.

Heidy Khlaaf of the AI Now Institute calls the extinction framing "fear-mongering" and points instead to AI's "low reliability and accuracy rates in critical environments with life-or-death consequences."