Maryland's β-OPSD lifts Qwen3-1.7B math up to 9.16 points

A quiet paper out of the University of Maryland lands on a question that has been nagging anyone actually trying to train reasoning models with on-policy self-distillation: why is the technique so brittle in practice? The paper by Liu, Zhang, Goldstein and Huang argues the answer is structural. Vanilla OPSD is not the whole objective, it is the β=1 case of a broader KL-regularized policy-optimization family, where β weights how tightly the student is anchored to a reference policy.

That reframing turns β from an implicit constant into a knob. The authors derive the closed-form optimal policy for the family and show it is a geometric interpolation between the reference policy and the privileged teacher. Directly optimizing that with RL would be costly and high-variance, so they use the derivation to specify the target, then approximate it locally by mixing reference and teacher logits at each token. A bounded linear schedule (w_start=0.5, w_end=0.8 over K=200 steps) moves the target from the reference toward the teacher over training, and a return-to-go weight with γ=0.99 propagates future mismatch back to earlier tokens.

Why this matters if you are not writing RL code: on-policy self-distillation is one of the cheaper paths to lift a reasoning model without full RL machinery, and the reliability tax has been real. Making β explicit and schedulable gives teams a principled dial instead of an implicit fixed choice, and the trick of using policy optimization to derive a target while training with distillation is a clean pattern others can borrow. On competition math benchmarks with Qwen3-1.7B, β-OPSD improved avg@12 by up to 9.16 percentage points over vanilla OPSD across AIME 2024, AIME 2025 and HMMT 2025.

The honest caveats are that the gains are not uniform. On Qwen3-4B, β-OPSD actually drops 1.11 points on AIME 2024 relative to vanilla OPSD even as the three-benchmark average rises 1.76. What the reporting does not give you is behavior beyond math reasoning, or beyond Qwen3 at scales past 8B, or a full sensitivity picture around the schedule endpoints. The forward-looking bit worth watching is whether this derive-with-RL, train-with-distillation pattern generalizes past math into code, agentic tasks and other verifier-rich domains where privileged information is available during training.