techpolicy.press web signal

Salvaggio: Code gains don't prove recursive self-improvement

TL;DR

  • Salvaggio argues gains on code and math benchmarks don't prove recursive self-improvement because reinforcement learning with verifiable rewards targets tasks with checkable answers, not open-ended research.
  • An Anthropic-commissioned METR report attributed, 'with substantial uncertainty,' 80% of last year's benchmark acceleration to AI code assistance, and METR says Anthropic redacted the most useful evidence.
  • In a summer paper, AI agents given unpublished NeurIPS 2026 research questions 'failed spectacularly,' unable to recalibrate when approaches became obviously fruitless.

The empirical case for recursive self-improvement, the idea that AI systems will rewrite themselves into an "intelligence explosion," is weaker than the loudest predictions suggest. That is the argument Eryk Salvaggio makes in an essay for Tech Policy Press published the day before Anthropic is slated to launch its initial public offering at a $1.4 trillion valuation.

Salvaggio opens in 1965 with mathematician Irving John Good, who worked alongside Alan Turing and defined the "ultraintelligent machine" as one that could "far surpass all the intellectual activities of any man, however clever" and design "even better machines," producing "an 'intelligence explosion,' and the intelligence of man would be left far behind." The modern version, he notes, has been voiced by Dwarkesh Patel, who predicted the world is "a couple of months away ... from AIs that are as good as the best humans at AI research," and, in turn, AI "at least as good the best human experts in every field" by 2030, or earlier.

Salvaggio's counter rests on two facts. First, an Anthropic-commissioned report from METR estimated that Anthropic's benchmark scores improved last year at a rate that would have taken two years in the years before, and attributed, "with substantial uncertainty," 80% of the acceleration to AI code assistance. METR also reports that the most useful evidence behind that estimate was redacted by Anthropic. Second, a summer research paper handed AI agents research questions drawn from unpublished NeurIPS 2026 submissions, with the original papers withheld. The agents, the authors write, "failed spectacularly," unable to recalibrate when an approach wasn't working.

"[A]gents can't hill-climb their way into choosing a set of candidate hypotheses, deciding what evidence would settle a research question, or incorporating feedback effectively and recognizing that an approach has failed and the right move is to start over," the authors wrote.

Salvaggio's through-line is that reinforcement learning with verifiable rewards works well on code and math because those tasks have checkable answers, and those gains do not translate to picking the right research question. "every improvement loop is a claim that some signal can substitute for human judgment," he writes. He closes on HAL 9000: Good himself consulted Stanley Kubrick on the 1968 film 2001: A Space Odyssey, and the machine is called super intelligent because it spoke and listened, even though the plot turned on the fact that it broke.

Shared on Bluesky by 2 AI experts