OPRD Paper Lets Student LLMs Surpass Weak Teachers Faster

A student model can surpass its weaker teacher, in fewer training updates than verifier-only reinforcement learning, if you treat the teacher's policy shift as a direction rather than a target. That is the pitch behind "Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation," posted to arXiv on September 8. The authors describe OPRD as a method that "evaluates the teacher's policy shift relative to its reference policy on student rollouts and amplifies the component of the student's verifier-driven policy gradient along that direction."

The motivation is a familiar failure mode: ordinary distillation "treats the weak teacher as an optimization target, potentially imposing its capacity ceiling on the student." By "rescaling only verifier-supported updates," the paper argues, OPRD "preserves the stationary points of policy optimization while accelerating learning beyond the teacher."

Response-style analysis found OPRD students "remain closer to models trained with verifier-based RL alone than to their weak teachers." The authors read that as evidence "teacher guidance accelerates rather than redirects the student's own optimization." The 38-page paper's abstract publishes no per-benchmark numbers or base-model names, leaving the size of the win over existing RL and distillation baselines unquantified in the public listing. It lands amid a run of post-training work we have tracked, 37 fine-tuning stories in the last 90 days, including IBM's DRACO on dynamic rubrics last week.