Evo-Bench measures if LLMs can rewrite their own agent harness

A team from Renmin University of China and BOSS Zhipin has put a number on something the agent research crowd has been circling for a year: whether a language model can actually improve its own scaffolding, not just the answers it produces. Their paper on Hugging Face introduces Evo-Bench, described as the first benchmark built to isolate a model's capacity for harness evolution across Search, Office, and General agent domains.

The setup fixes DeepSeek-V4-Flash as the policy model and starts every run from the same CodeAct seed harness, giving each evolver a common 48-hour, 20-iteration, 1,000-step budget. Across nine frontier and open-weight models, GPT-5.6-Sol tops the leaderboard at 46.3 overall (+16.6 over CodeAct), with Claude Opus 4.8 close behind at 45.8 (+16.1) and GLM-5.2 at 43.5. The composite human-engineered Artificial harness scores 47.5, so the best autonomous evolvers close most of that gap without quite matching it.

The per-domain breakdown is where the story sharpens. On Search, Claude Opus 4.8 gains +34.8 to 46.5, effectively parity with the 46.7 human baseline. On General tasks, GPT-5.6-Sol and Qwen3.7-Max both hit 59.4, strictly beating the Artificial harness's 56.3, which the authors call a milestone for autonomously synthesized reasoning structures. Office is the sore spot: top score of 41.6 trails the 43.9 human baseline, and most evolvers show marginal gains or slight regressions, which the paper attributes to the difficulty of discovering specialized workflows. This is now the third harness-and-agent benchmark we've written up in as many days, alongside SWE-Bench ProMax and Antigma's Terminal-Bench run, which is a decent signal that harness-level evaluation is where the field is heading.

Cost is the other headline. GPT-5.6-Sol's evolver runs exceed $500 apiece, while GLM-5.2 and Qwen3.7-Max sit under $40 for competitive scores, and DeepSeek-V4-Pro produces functional harness improvements for less than a dollar. The paper also flags early saturation as a real behavior: Qwen3.7-Max and DeepSeek-V4-Pro halt around 15 iterations using barely 200 steps, well short of the maximum budget.

A few things the paper does not resolve. It runs each experiment only once, so single-run variance is unmeasured, and it never tests whether the evolver ranking survives swapping the fixed DeepSeek-V4-Flash policy or the Qwen3.7-Plus judge. For teams building agent products, the takeaway is narrower than 'models can now build themselves': autonomous harness evolution is competitive on open-ended reasoning, workable on search, and still needs a human on the office-workflow side.