Evo-Bench tests if LLMs can improve their own agent harness

A new benchmark paper on arXiv, Evo-Bench: Can Language Models Improve Agent Harness?, asks a question that has been floating around agent research for a while: can a model do a decent job of rewriting the scaffolding it runs inside, rather than just answering the task in front of it? The authors, led by Lisheng Huang and colleagues including Wayne Xin Zhao, position it as the first benchmark aimed squarely at that harness-evolving capability, across three agent domains they call Search, Office, and General. Two experts in our Who's Who directory have already shared the preprint.

The headline result is that across nine frontier and open-weight models, top performers post absolute gains reaching 16.6 points, and the paper claims those gains close in on state-of-the-art human-engineered baselines. On General and Search tasks, autonomous evolution actually beats the human-designed harnesses; on Office tasks, which the authors describe as demanding highly specific processing workflows, it struggles. The paper also argues that the synthesized harnesses behave as highly transferable reasoning structures, consistently boosting diverse policy models rather than only helping the model that produced them.

The construction is where the paper spends its methodological credit. Rather than a static task set, Evo-Bench uses what the authors call a harness-guided construction framework, leaning on auxiliary-task evolution to pick tasks that are genuinely sensitive to framework improvements, then a sensitivity-aware stratified split to guard against cross-suite overfitting. That is a direct response to the standing complaint that agent leaderboards mostly measure base-model strength dressed up as agent skill.

A few things the abstract does not settle. It does not name which of the nine models hit the 16.6-point top gain, nor what the specific human-engineered baselines it is chasing actually are. The authors also flag temporal anomalies like early saturation during autonomous optimization without quantifying how quickly that plateau kicks in, which is exactly the kind of failure mode that matters for anyone thinking about long-horizon research agents.

If the transferability claim survives outside evaluation, the practical upside is not that models replace human harness engineers overnight; it is that a good synthesized harness in one lab could lift a whole family of policy models, which is a very different economic story than bespoke scaffolding per deployment.