Nature weighs the science behind the AI extinction panic

A researcher at Anthropic quit on September 8 and posted that "the people building AI earnestly believe that it could kill us all by the end of the decade." Jacob Coxon's note crossed 100 million views in 24 hours.

Evan Hubinger, who leads alignment science at the same company, amplified Coxon and put "the risk of human extinction is >10% within the next decade."

In Nature, Elizabeth Gibney walks through what the underlying science actually supports. The case rests on two assumptions: that AI will one day exceed human intelligence, and that its goals will not match ours. From there come the stock pictures. The "paperclip maximizer" optimises a single objective until nothing else is left. A more vivid version has an AI that "unleashes a biological weapon to kill humans off and thereby make more space for solar panels and robot factories."

Michael Vermeer, who researches science and technology policy at RAND Corporation, is unmoved. The extinction scenario "involves so many untestable claims that you just end up with a conversation that is really more like faith than something scientific," he tells Nature. A 2025 RAND study found complete nuclear extinction infeasible; biotech and atmospheric routes could not be ruled out, but either would require significant physical capabilities and would take time in which people might intervene.

Heidy Khlaaf, chief AI scientist at the AI Now Institute, calls the existential framing "fear-mongering." She keeps her focus on what AI is already doing badly: disinformation, psychosis enablement, bioweapon help, military mishaps.

The piece also notes who gains. Anthropic's Dario Amodei, OpenAI's Sam Altman and xAI's Elon Musk have each called for a development slowdown, and nearly 1,400 AI-sector employees have signed an open letter asking for the same. A safety-first regulatory frame tends to suit incumbents. David Sacks warned that firms face "massive product-liability exposure" from damaging cyberattacks enabled by their models. Four of the researchers in our Who's Who directory were among those circulating the story.