W2S-OPD Distills Stronger LLMs Using Weaker Teacher Pairs

The awkward problem at the top of the model scaling curve is that distillation, the standard way you cheaply transfer skill from one model into another, assumes a teacher at least as capable as the student. Once you are the biggest model in your family there is no bigger teacher, and training a fresh expert at student scale to consolidate into is expensive. A new paper from a group led by Fangxu Yu, posted to arXiv in late July 2026, proposes a way around that.

The method, called Weak-to-Strong On-Policy Distillation or W2S-OPD, takes two models that are both smaller than the student, a 'positive' and a 'negative' pair, and uses the difference in their logits to isolate what the authors call the 'capability direction'. That direction gets added on top of the student's own base model in logit space, producing a proxy teacher that carries the desired skill signal while staying, in the authors' words, 'distributionally adjacent to the student'. The student then trains on its own rollouts and minimizes the per-token reverse KL to that proxy.

They instantiate the contrast pair three ways: a post-RL expert against its pre-RL initialization to isolate what RL taught it, a larger base model against a smaller one to isolate the effect of scale, and a small base model given correct versus wrong hints to isolate the direction toward the answer on a specific problem. The reported claim is that across four math benchmarks and three code benchmarks, W2S-OPD beats standard on-policy distillation, lets the student surpass its domain teacher, and keeps improving the student 'even when every supervision source is weaker'.

The honest caveat is that the abstract is where most of the specifics live in what I could retrieve. It does not name the individual benchmarks, does not put a number on the gap over standard OPD, and does not tell you how sensitive the method is to the size gap between the weak pair and the strong student. It also only tests on math and code, the domains where you can grade a rollout cleanly, which is exactly where reverse-KL-on-rollouts recipes tend to do well.

Still, the direction is the interesting part. If a strong model can genuinely be improved by contrasts of smaller models it already has lying around, the training-cost picture at the frontier shifts, and the throwaway intermediate checkpoints most labs already produce start looking like reusable supervision instead of exhaust.