Study: Agent Harness Evolution Doesn't Beat Test-Time Scaling

A new arxiv paper takes aim at a fashionable claim in agent research: that letting an LLM automatically evolve its own harness, the scaffolding of prompts, tools and control flow around the model, is why agent benchmark scores keep climbing. The authors argue those gains may be an artefact of how the systems are evaluated, not evidence of a better harness.

Their paper on arxiv, 'Rethinking the Evaluation of Harness Evolution for Agents,' lodges two methodological complaints. First, harness evolution is itself an iterative search procedure that repeatedly evaluates and revises candidate harnesses using task feedback, so to know whether its gains come from better design or just from spending more compute on search, it should be compared with plain test-time scaling under matched feedback and inference budgets. Second, because the search and the final evaluation share the same benchmark, the reported wins risk overfitting to that specific task set.

The team then runs that comparison. On Terminal-Bench 2.1 with GPT-5.4 and Claude Opus 4.6, they report that automatic harness evolution does not consistently outperform simple test-time scaling methods, and that evolved harnesses exhibit limited generalization on held-out tasks. The framing is a call for fairer evaluation protocols and benchmarks for automatic harness design, not a claim that the underlying idea is worthless.

The honest caveat is that the abstract does not publish the size of the gap or name the exact test-time scaling baseline, so how large the effect is matters for anyone trying to price the finding into a roadmap. What the reporting does not give you is whether GPT-5.4 and Claude Opus 4.6 diverged under this scrutiny, which would change the conclusion for teams standardized on one vendor.

For engineering leaders shipping agents, the useful move is smaller and more boring than the headline. Before paying up for an 'auto-evolving' harness stack, ask the vendor for held-out task numbers and a matched-budget test-time-scaling baseline. If the delta collapses under those conditions, so does the case for the premium.