Elo-per-Token Study: Parallel Sessions Beat Long Agent Runs

For four general-purpose LLM agents tested on four open-ended benchmarks, early tokens produce more skill per unit of compute than random independent sampling. But that lead shrinks as sessions grow, and eventually the agents' marginal gains fall below the random-sampling reference.

That is the finding of "When Agents Slow Down", a preprint from Kaiyuan Liu, Qiuyang Mang, Luke Zettlemoyer, Alex Dimakis and collaborators. Their proposed method, "Elo-per-token analysis," tracks the best solution found at each token budget and uses a Bradley-Terry model to aggregate within-task orderings into Elo ratings across tasks with different score scales. Independent sampling provides a theoretically characterized reference "for which Elo grows linearly with log compute."

Humans on the same tasks do the opposite. "The strongest historical human contestants improve superlinearly over contest time on shared AtCoder Heuristic Contest tasks," the authors write, calling it "evidence of continual learning and substantial headroom after agents slow down."

The paper defines a "scaling inflection point," the per-session budget where marginal Elo gains match the independent-sampling reference. Using that point as the split, the authors broke a 100M-token budget across parallel sessions on FrontierCS Polyomino Packing and reported +264 Elo over one long session and +355 Elo over ten short sessions.

Agent-methodology work has been dropping in clusters lately; this one arrives alongside RSIAgent and Shanghai AI Lab's Atria Dawn Preview on our tracker for the same day.