Study: 82% of outcome RL gains are finishing, not reach

Answering before the token budget runs out, which the paper labels 'finishing', accounts for 82% of pass@1 change at 4k and 8k token limits across 20 released reinforcement-learning models tested on physics and math. That is the central finding of a new arxiv preprint comparing outcome-RL models of 1.5B to 32B parameters with the thinking models they started from.

"Reach adds at most 2.1 percentage points on any of our four physics and math sets," the paper reports. Reach, in its definition, counts problems the RL model solves that its base missed in all eight tries, net of problems the base solves that RL missed in all eight.

The author pulls 144 physics problems solved only by the RL model and rechecks them. "Of these, 3.5% hold up when resampled or read." A random sample of what looked like novel RL reasoning steps goes the same way: "nine of ten new RL steps a human confirmed also appear in the base's samples."

Give the base model enough tokens and most of the finishing gap closes itself. Where tokens are abundant, the RL model's edge is reliability: it is "graded right more often, even on problems both finish." In a small-scale RL run on an 8B model, "the gain on new problems is finishing, with reliability flat."

Supplying each problem's physical setup to the model was not shown to improve on plain RL. The analysis builds on Yue et al., 2025, which found outcome RL raises pass@1 but not pass@k at large k.