SJTU's EarlyEval Halts Agent Runs Early to Cut Eval Costs

Researchers from Shanghai Jiao Tong University and Singapore Management University want to stop LLM-agent benchmark runs the moment a lightweight classifier is confident the run will succeed or fail. Their EarlyEval paper reports that on SWE-bench Verified, TerminalBench and Toolathlon, the method eliminates 13% to 26% of agent steps and up to 44.1% of input tokens and 29.4% of output tokens at 89% to 97% prediction accuracy, while perturbing per-agent resolve rates by only one to two percentage points on average.

The machinery is deliberately small: a pair of LightGBM success and failure classifiers trained over behavioral, textual and reference-solution features, run at every step of a new agent's rollout. "The moment either classifier crosses a predefined confidence threshold, we terminate the run and record the predicted outcome," the authors write. Training data comes from more than 21,000 outcome-labeled trajectories drawn from 16, 37 and 22 distinct agents on the three benchmarks, with a leave-one-agent-out protocol so the classifier judges an agent it has never seen.

The cost pressure the paper leans on is real. Citing the OpenHands Index, the authors note a single frontier-model pass on SWE-bench Verified "costs several hundred dollars," with longer-rollout benchmarks "reaching into the thousands of dollars per pass," exceeding $2,200 for the most costly model in their Table I. Prior benchmark-distillation work shrinks the task list but, as they put it, leaves "the per-task execution cost untouched."

On SWE-bench Verified specifically, EarlyEval halts roughly 35% of runs at 95% prediction accuracy, eliminates 26% of execution steps along with 33% of input and 29% of output tokens, and reproduces the full-run leaderboard at a Spearman rank correlation of 0.991 across all 16 agents, with only three adjacently ranked agents shifting by a single rank. It arrives alongside a run of agent-tooling releases we have tracked this week, including BAAI's DisCo skill-distillation paper, one of more than 400 agent stories we have covered in the last 90 days, and the authors pitch early outcome prediction as an axis of efficiency that complements, rather than replaces, task-set distillation.