Anthropic alignment lead puts AI extinction risk above 10%

AI extinction forecasts rest on "so many untestable claims that you just end up with a conversation that is really more like faith than something scientific or empirical," Michael Vermeer, a science and technology policy researcher at RAND, tells Nature in a feature by Elizabeth Gibney testing the empirical case behind this autumn's wave of extinction warnings.

The warnings traced back to September 8, when Anthropic researcher Jacob Coxon resigned and told the Wall Street Journal he feared the company's systems "could spiral out of control and destroy humanity." Evan Hubinger, who leads Anthropic's alignment science efforts, then put the "risk of human extinction" at ">10% within the next decade." The post drew over 100 million views in 24 hours. Anthropic CEO Dario Amodei followed with a call to slow, not halt, AI development, a position Sam Altman at OpenAI and Elon Musk at xAI endorsed.

The science Nature consulted is messier than the forecasts suggest. A 2025 RAND study concluded that engineering complete human extinction through nuclear weapons is infeasible, while routes through biotechnology or atmospheric modification cannot be ruled out, though both would require an AI to physically interact with the world and would likely be detectable. Heidy Khlaaf, chief AI scientist at the AI Now Institute, calls the extinction framing "fear-mongering" and points instead to AI's "low reliability and accuracy rates in critical environments with life-or-death consequences."

The nearer harms Nature catalogs are the ones already landing: disinformation, psychosis, assistance with bioweapon creation, cybersecurity exposure, and a CNN-reported episode of AI-generated false intelligence that nearly triggered a US military incident with a Chinese ship. Nearly 1,400 AI-sector employees have signed an open letter calling for a slowdown.

Industry economics shade the debate. Anthropic is reportedly preparing an IPO, giving it room to argue publicly for slower development without ceding competitive ground. David Sacks, co-chair of the US President's Council of Advisors on Science and Technology, flags "massive product-liability exposure" from damaging cyberattacks as the more immediate corporate concern. Amodei himself points to recursive self-improvement, meaning AI systems building successor systems, as the control problem that worries him most.