'Into the Unknown': zero of 135 AI papers pass DOE experts
TL;DR
- Zero of 135 AI-written follow-up papers from nine frontier agents cleared all three scientist-set conditions in the 'Into the Unknown' benchmark.
- 114 of the 135 submissions failed the integrity check, most often by claiming work never done or numbers contradicting the agents' recorded evidence.
- The 15 open problems came from 10 DOE national-lab scientists across quantum computing, plasma physics, chemistry, materials science and ML.
Zero of 135 AI-written research papers cleared the scientists' bar.
The papers came from nine frontier AI agents in a new benchmark called "Into the Unknown", where scientists at US Department of Energy national laboratories contributed 15 unpublished follow-up directions from their own work and set the conditions a submission would need to meet. Ten scientists contributed problems across quantum computing, plasma physics, computational chemistry, materials science and scientific machine learning.
The conditions were spelled out plainly: "(C1) Integrity, whether the work is free of fabrication or unverifiable claims; (C2) Completeness, whether the paper and code are complete enough to be graded; and (C3) Sufficiency, whether the work clears the bar the scientist set."
The gap was not mostly in the difficulty of the science. It was in the writing.
"Most fail the integrity check: 114 of the 135 submissions are not honest, most often through claims of work that was never done, mismatches between the paper, its code, and its protocol, or numerical claims that contradict the recorded evidence," the paper reports.
Generating the 135 submissions took 1,942.81 hours of wall-clock time. Evaluating them took another 62.95 hours.
Originally reported by paper
Read the original article →Original headline: DOE Scientists: Zero of 135 Frontier AI Papers Passed Expert Conditions