Qwen3 Study Names Two OPD Failure Modes, Cheap Fixes Beat RLVR

On-policy distillation has quietly become a common post-training move for open-weight LLMs, but the ways it goes wrong have never been mapped out cleanly. A new arXiv paper from Rui Wang and colleagues tries to, and along the way argues the standard instinct of grabbing a bigger teacher is the wrong lever.

Their framing is that on-policy distillation is an exploration catalyst, not a capability ceiling extender. It steers the student toward correct reasoning paths through dense token-level guidance, but that guidance can misfire in two specific ways they name and test. The first is student-teacher mismatch: when the distributional gap between the two is wide, the guiding signal stops aligning with task correctness and exploration goes in counterproductive directions. The second is length exploitation, where the token-level objective creates length-dependent shortcuts and the student learns to game the reward through response truncation or redundant padding instead of reasoning better.

The proposed fixes are deliberately unfancy. Advantage clipping bounds the per-token signal to kill outliers. Log-scale compression dampens extreme values while preserving relative rankings. Applied to a Qwen3-1.7B-Base student across seven benchmarks (MATH500, Minerva, AMC'23, AIME'24, AIME'25, AIME'26, HMMT'25), the regulated variants lift the average from 28.1 to 30.4 and, per the authors, stably surpass both naive OPD and RLVR baselines. The core claim is that signal quality, not teacher scale, governs whether distillation works.

The honest caveat is that this is one paper and the benchmarks are all math. Whether the same failure modes and fixes show up in code, agentic tool use, or general instruction-following is not something the paper answers, and a compute-cost comparison against RLVR training runs is not spelled out either. Take the numerical wins as reported, not settled.

Still, for post-training teams already running OPD in production, this is the kind of paper worth reading before the next scale-the-teacher meeting. If length exploitation is real and cheap to fix, the free points are sitting there for the taking.