Nature weighs the science behind AI extinction warnings

A pretraining researcher resigned from Anthropic on September 8 and told the Wall Street Journal he feared the systems his employer was building could spiral out of control and destroy humanity. In Nature's telling, that is where the latest round of extinction discourse took off: Jacob Coxon's post went viral, and Evan Hubinger, who leads Anthropic's alignment science, reposted it with his own estimate that the risk of human extinction is '>10% within the next decade.'

Anthropic chief executive Dario Amodei has since called for a slowdown, not a halt, in AI development. Sam Altman of OpenAI and Elon Musk of xAI backed the suggestion. Nature then does something the discourse rarely does, which is ask what the mechanism would actually be. The piece surfaces the familiar thought experiments: the 'paperclip maximizer' that renders Earth uninhabitable in pursuit of its goal, and the speculative 'AI 2027' forecast in which an AI unleashes a biological weapon 'to kill humans off and thereby make more space for solar panels and robot factories.'

Michael Vermeer, a science-and-technology policy researcher at RAND, is unimpressed. The extinction case, he tells Nature, involves 'so many untestable claims that you just end up with a conversation that is really more like faith than something scientific or empirical.' Heidy Khlaaf, chief AI scientist at the AI Now Institute, calls the framing 'fear-mongering' and points at harms already landing: disinformation, bioweapon support, and the AI-generated false intelligence that Nature says recently nearly triggered a naval incident involving a Chinese vessel.

Four researchers in our Who's Who directory shared the Nature link, which fits the shape of the piece: it does not settle whether AI will kill us all. It sets Hubinger's '>10%' next to Vermeer's 'faith' and lets the reader sit with the gap.