β-OPSD adds one knob to self-distillation, lifts Qwen3-1.7B math by 9.16pp

Most of the recent progress in post-training reasoning models has come from scaling teachers, curating harder data, or moving to full reinforcement learning. A recent arXiv preprint from Jiawei Xu, Minghui Liu, Juzheng Zhang, Tom Goldstein and Furong Huang makes a smaller and, I think, more useful kind of claim: the dominant cheap alternative, on-policy self-distillation (OPSD), has a specific structural bug, and there is a one-parameter fix that costs almost nothing to add.

The argument is that vanilla OPSD is precisely the β=1 member of a broader policy-optimization family, where β weights the KL penalty anchoring the student to a reference policy. Turning β from an implicit value fixed at one into a controllable knob lets you trade off proximity to the reference model against privileged teacher guidance. The optimal policy under that objective works out to a geometric interpolation between reference and teacher, which the authors implement by mixing the two models' token-level logits. No extra RL loop, no new learned parameters, and a return-to-go credit-assignment tweak that keeps token updates aligned with the sequence-level objective.

On Qwen3-1.7B, the reported numbers are 9.16, 5.27 and 2.78 percentage-point gains over vanilla OPSD on AIME 2024, AIME 2025 and HMMT 2025 respectively, for a 5.74-point average improvement on avg@12. The authors report consistent gains across three Qwen3 sizes, with the strongest effects on the smallest model.

The honest caveats are that the headline number is on the smallest model and the easiest of the three benchmarks, the paper is fresh preprint work without independent replication, and β is now a hyperparameter that has to be tuned. What the reporting doesn't give you is whether the fix carries beyond competition math into code, agents, or the frontier-scale reasoning systems everyone is currently training with heavier RL.

If it holds up, the practical read is simple: teams already running OPSD on reasoning models get an almost-free lift by replacing a token-level target with a logit mix, before paying for bigger teachers or a full RL pipeline.