HarnessOpt-Bench Finds Optimizer LLM Beats Harness Choice

A new arxiv paper puts a number on a question every team building LLM agents has been arguing about: when you try to squeeze more out of an agent by iterating on its prompts, tools, control flow and orchestration code, does it matter more which model you use to do the iterating, or which coding harness the iterating happens inside?

The paper, HarnessOpt-Bench, tries to answer that with a repeatable protocol. An optimizer, defined as an LLM paired with a coding harness, receives a target agent's seed harness, graded evaluation feedback, and a fixed target-evaluation budget. It edits the harness and nominates a final candidate, which is scored on a held-out test partition that remains inaccessible throughout search. A trusted execution environment enforces that boundary, meters resource use, and preserves candidate versions for audit. Five frontier LLMs were run as optimizers, both under a shared coding harness and under their native harnesses, across four downstream tasks, for 111 scored runs in total.

The headline finding is that optimizer models separate more than the coding harnesses they act through. In plainer terms, the LLM you point at the problem moved results more than the framework you wrapped it in. The paper also reports that native harnesses are not consistently superior to the shared one, and that gains vary substantially across tasks and seed regimes.

Why this matters if you are picking agent tooling: a lot of the current vendor pitch is that a proprietary harness is the special sauce. A result like this suggests that claim needs a benchmark next to it, not a demo, and that model procurement and harness procurement should be treated as one decision, not two. It also raises the possibility that some of the intra-lab gap on agentic tasks is really a gap in each model's ability to improve its own scaffolding.

The honest caveat is scale. Four downstream tasks and 111 runs is a first data point, not the last word, and the authors themselves flag that gains vary by task and seed. What the paper does not disclose in the abstract is which specific five frontier models were tested and how they ranked individually, or what the four target tasks were, so the shape of the effect is public but the labelled leaderboard is not. If future versions expose those rankings, the benchmark becomes a real buying signal for anyone standardizing on an agent framework.