Infrathon benchmark: 135 of 150 agent runs keep improving

Of 150 runs on a new long-horizon AI infrastructure benchmark, 135 kept improving during the second half of their 24-hour budget.

That is the central claim of Infrathon, a paper from a group that includes Noah A. Smith, Hannaneh Hajishirzi, Dan Roth, and Radha Poovendran. The authors evaluated six models across 150 runs with 24-hour budgets, "totaling over 10,000 GPU-hours."

"Our results show that model rankings vary with the evaluation horizon, while 135 of the 150 runs achieve further improvements during the second half of their budgets," the abstract states. "These findings indicate that shorter evaluations can overlook both optimization gains and changes in relative model performance."

One concrete example of an agent beating a human baseline: Claude Opus 5.5 "increases the mean KV-cache compression ratio from 6.82% with the human-designed Quant-VideoGen codec to 19.86%."

The authors flag failure modes too. Analysis of agent trajectories "identifies task violations, unreliable testing, and premature completion as barriers to sustained improvement." The abstract publishes no rates for the individual failure modes, and no per-model leaderboard at different horizon lengths.