Direct-OPD Ports Weak-Model RL Gains to Qwen3-1.7B on AIME

The interesting bit in this new arxiv paper is not the AIME score, it is what the score implies about where post-training costs are heading. A Qwen3-1.7B student reportedly went from 48.3 percent to 58.3 percent on AIME 2024 in four hours on eight A100 GPUs. The reinforcement learning that produced those gains was not run on the 1.7B model at all in the usual sense. It was run on a weaker teacher, and only the shift between the teacher's pre-RL and post-RL checkpoints was transferred over.

The authors call the method Direct On-Policy Distillation. Their framing is that RL with verifiable rewards keeps getting more expensive to repeat because the target model has to generate many rollouts during training, and rollouts scale with the model you actually care about. So they run RL cheaply on a small model, compare that model's before and after weights, treat the log-ratio between them as what they call a dense implicit reward, and apply that signal on the stronger student's own on-policy states. Naively distilling from the fine-tuned weak teacher does not work, the paper argues, because you inherit the small model's limitations along with its RL gains.

Why this matters if you are not training frontier models yourself: post-training has been quietly becoming the bottleneck for anyone shipping reasoning models, and the default today is to redo the RL run on each new base checkpoint. If a policy-shift transfer like this holds up outside the specific setup shown, the economics get better for open-weights labs and small research groups that cannot afford to rerun RLVR on every new base.

The honest caveat is that the reported result is one student, one benchmark, one paper. The abstract discusses Qwen3-1.7B on AIME 2024, and it does not tell you how the method behaves on larger students, how sensitive it is to the choice of teacher, or how it compares to running RL directly on the target if you could afford to. Take the ten point jump as promising evidence, not a settled recipe. What is worth watching over the coming months is whether independent groups can reproduce the transfer at larger student sizes, because that is where the compute bill actually bites.