LoopArena benchmark caps AI controller success at 24.69%

The best Controller model in LoopArena reached only a 24.69% Strict Success Rate when directing a fixed coding agent through complete tasks, according to a new arXiv paper introducing the benchmark.

LoopArena splits the roles on purpose. The model under evaluation is the Controller: after each coding round, it reads a structured summary and tells a separate, fixed Worker agent what to do next or whether to stop. The authors argue this separation is the whole point, because "the final outcome of one end-to-end run cannot tell whether success or failure reflects the loop's guidance or the coding agent's ability to carry out the task."

Three evaluation modes trade cost for realism. Type I scores next-step decisions with execution-validated questions and never runs the Worker. Type II runs the Worker over a slice of a full task. Type III runs the paired full task from its original state. Across Controllers, Type II tracked full-task rankings tightly, with the paper reporting "Spearman's ρ=0.9747," and the paired reduction in estimated inference cost averaged 64.4%.

The paper frames loop engineering as an emerging practice in which developers "design loops that monitor progress, assign work, run checks, and decide what the agent should do next," but where a loop can still fail if the Controller trusts a stale progress note, skips verification, spends its budget in the wrong direction, or stops before the task is safe to submit.

The abstract does not name which Controllers were tested, gives no per-model breakdown, attaches no dollar figure to the 64.4% cost gap, and offers no comparison against a naive fixed-schedule controller. Data and evaluation code are released at github.com/AMAP-ML/LoopArena.