HarnessDev: LLM-built harnesses lag on code, match writing

When frontier models are asked to build the execution scaffolding for their own agent runs, the results split cleanly by domain. That is the finding of HarnessDev, a benchmark described in a September 2026 arXiv preprint covering six creator LLMs, four domains, and 2,207 downstream test instances. Generated harnesses "remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation."

The paper reframes what an agent benchmark measures. Rather than scoring task outputs under a fixed harness, HarnessDev shifts "the unit of evaluation from task outputs to runnable infrastructure." Its Creation stage hands a model a minimal seed and a small case set and asks it to build the execution system. Its Evolution stage lets the model iterate on its own harness using downstream feedback.

Neither stage is a clean win. The authors report that "Evolution produces some performance gains, but they are unstable and transfer only partially to held-out tasks." A separate experiment fixing the runtime model found the gains "depend strongly on the model executing the harness, indicating limited transfer across models." The abstract notes "large variation in execution cost" across generated harnesses but publishes no per-domain figures.