CalibForge Adds 27.68pp on SWE-bench Pro via Solver Calibration

A quietly interesting arxiv paper dated August 6, 2026 argues that the bottleneck for training terminal agents is not the model, it is the training data, and specifically whether that data sits in a difficulty band a solver can actually learn from. The authors, in a paper titled CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks, describe a synthesis system that uses verified solver behavior to revise candidate tasks, keeping only the ones that sit in what they call a solver-relative learnable zone anchored in demonstrated solvability.

The mechanic is worth understanding. Instead of authoring tasks and validating they run, CalibForge runs a heterogeneous pool of solvers against candidates and keeps those that produce useful disagreement (multi-solver calibration), or where a strong solver passes and a weak solver fails (contrastive solver calibration). The output is a set of 5,431 calibrated terminal tasks, and the paper's ablations claim both strategies beat plain authoring, validation, or single-solver feedback.

The headline numbers, reported by the authors, are the reason to pay attention. Models trained on the full collection reach 32.58% and 47.57% on Terminal-Bench 2.0. The largest improvements over the corresponding base model reach 24.71 percentage points on Terminal-Bench 2.0, 27.68 points on SWE-bench Pro, and 30.04 points on Doc2Repo. That is benchmark movement that would normally read as a scaling win, coming instead from a data recipe.

The honest caveat is that this is a single preprint with self-reported ablations, and the abstract does not tell you which base models were fine-tuned, how the evaluations were run, or how the gains distribute across task types. Take the specifics as reported, not settled.

What is worth watching, if the results hold up, is where the competitive advantage lands. If solver-relative synthesis really pulls this much out of existing base models, the teams best placed to benefit are the ones that already operate heterogeneous evaluator fleets and can afford to burn compute on adversarial filtering, rather than the ones with the largest checkpoints.