β-OPSD adds one knob to self-distillation, lifts Qwen3-1.7B math by 9.16pp

Most of the recent progress in post-training reasoning models has come from scaling teachers, curating harder data, or moving to full reinforcement learning. A recent arXiv preprint from Jiawei Xu, Minghui Liu, Juzheng Zhang, Tom Goldstein and Furong Huang makes a smaller and, I think, more useful kind of claim: the dominant cheap alternative, on-policy self-distillation (OPSD), has a specific structural bug, and there is a one-parameter fix that costs almost nothing to add.

The argument is that vanilla OPSD is precisely the β=1 member of a broader policy-optimization family, where β weights the KL penalty anchoring the student to a reference policy. Turning β from an implicit value fixed at one into a controllable knob lets you trade off proximity to the reference model against privileged teacher guidance. The optimal policy under that objective works out to a geometric interpolation between reference and teacher, which the authors implement by mixing the two models' token-level logits. No extra RL loop, no new learned parameters, and a return-to-go credit-assignment tweak that keeps token updates aligned with the sequence-level objective.

On Qwen3-1.7B, the reported numbers are 9.16, 5.27 and 2.78 percentage-point gains over vanilla OPSD on AIME 2024, AIME 2025 and HMMT 2025 respectively, for a 5.74-point average improvement on avg@12. The authors report consistent gains across three Qwen3 sizes, with the strongest effects on the smallest model.

A few things to keep in mind: the biggest gain lands on the smallest model and the easiest of the three benchmarks, the paper is a fresh preprint without independent replication, and β is now a hyperparameter that has to be tuned. The evaluation also doesn't show whether the fix carries beyond competition math into code, agents, or the frontier-scale reasoning systems everyone is currently training with heavier RL.

If it holds up, the practical read is simple: teams already running OPSD on reasoning models get an almost-free lift by replacing a token-level target with a logit mix, before paying for bigger teachers or a full RL pipeline.