Frontier Agents Handle Engineering, Fail NeurIPS Shadow Review

A new paper on Hugging Face, posted July 29 by Peter Kirgis, Sayash Kapoor and a long author list, makes a specific empirical bet about how far frontier AI agents actually are from doing research on their own, and the answer is not flattering. The setup, which the authors call a "shadow evaluation," is straightforward. Take two unpublished NeurIPS 2026 submissions. Hand the central open-ended research question to a frontier agent. Give it six days and thousands of dollars in compute. Then have the original human authors grade what comes back.

Both attempts were "unambiguously rejected by the authors." That is the headline result, and it is worth sitting with. The agents completed all the engineering work without human help. They just could not make substantial progress toward answering the research questions themselves.

The paper decomposes the gap into five recurring failure modes: poor judgment about what constitutes publishable research, uncreative responses to research shortcomings, ineffective backtracking from dead ends, poor resource awareness, and instruction drift. A second model and scaffold configuration reproduced the same failures, which is the part that argues the result is about the current generation of agents rather than one bad run.

Why this matters if you are trying to work out how quickly AI research itself gets automated: most of the loud public benchmarks measure narrow, verifiable tasks, which is exactly the layer the paper says agents already handle. The layer they don't handle is the harder, quieter one. Deciding whether a result is worth writing up. Noticing the experimental design has a hole. Choosing to abandon a doomed direction and pivot. Those show up in the five failure modes but not on most leaderboards.

The honest caveat is that this is two papers and one methodology, and the authors themselves frame it as early evidence. What the reporting doesn't give you is which specific frontier models were tested, how the two NeurIPS submissions were selected, or how consent was handled for using unpublished work this way. The forward-looking piece is that the authors released the expert reviews, survey responses, agent repositories, and execution logs, so the next round of scaffolding and post-training work has a real, human-graded target to shoot at rather than a vibes benchmark.