Paper: RLVR's Diversity Loss Sits at the First Token

Reinforcement learning with verifiable rewards narrows a model's solution space almost entirely at the first reasoning step, not across the long chain that follows, Qiancheng Zhou and Ruizhe Li report on arXiv. Per-token likelihood shifts, they find, are '11x--16x larger prior to the first arithmetic operation than during downstream reasoning.'

Studying the Countdown task with PPO on Qwen2.5-3B and GRPO on Qwen2.5-3B-Instruct, the authors measure solution coverage dropping by up to 67% after RLVR training. The alternative solution paths themselves are not destroyed. When the researchers hand the trained model an unused solution prefix and let it continue, completion rates jump from 0.018 to 0.212 under PPO. The routes stay executable; the policy simply no longer initiates them.

The paper, titled 'Locked at the Entrance, Open Inside,' offers a cheap intervention: interpolating late-layer parameters with an earlier training checkpoint. That step raised coverage by 37% 'without sacrificing pass@1 performance.' The early-step entropy collapse then reappeared across six math benchmarks and models up to 14B, though staged SFT-DPO-RLVR pipelines preserved more diversity.

The Countdown focus and the 3B-to-14B scale leave the frontier case open. What the study does establish is a specific location for a debate over test-time scaling that has been fought mostly at the level of aggregate pass@k curves.