New paper: sequential OPD-then-RL beats joint RLVR fusion
TL;DR
- A two-stage OPD-then-RL scheme beat pure OPD, pure RLVR, and every joint baseline the authors tested on logic and math reasoning benchmarks.
- Jointly optimizing on-policy distillation with RLVR causes the two signals to interfere rather than complement, per the paper's pass@k and learning-dynamics analysis.
- The OPD validation score is proposed as the switch trigger to RL, and OPD is reported as a better cold start than supervised fine-tuning.
The paper's central claim, from an arxiv preprint posted 3 September by six authors led by Boyan Li: "a simple two-stage scheme, OPD-then-RL, consistently outperforms pure OPD, pure RLVR, and all such joint baselines across logic and math reasoning benchmarks."
The two methods in question are RLVR (reinforcement learning with verifiable rewards) and OPD (on-policy distillation), which the authors describe as "two dominant methods for post-training reasoning LLMs." Prior work had fused them within a single training step, either as a weighted-additive combination or as a teacher-modulated rescaling of the RL advantage.
The mechanism the authors offer for why the sequential run wins: "OPD expands the student's coverage of teacher-supported solutions and RL sharpens within that support, while jointly optimizing the two signals causes them to interfere."
For practitioners, the paper proposes one operational switch trigger, the OPD validation score, and reports that OPD is "a better cold start for RL than SFT." Two of the researchers we track posted the preprint into their feeds within days of it going up.
The abstract publishes no head-to-head deltas against the joint baselines, and does not name the base models, teacher-student pairs, or the specific benchmarks used to reach the ranking.
Shared on Bluesky by 2 AI experts
Originally reported by arxiv.org
Read the original article →Original headline: Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR