NAVER's OPD² Distills the Reasoning Delta From Teacher LLMs

A quiet result out of NAVER this month is worth a beat: instead of teaching a student LLM to imitate everything a reasoning-tuned teacher says, teach it only the part the reasoning tuning actually added. That is the pitch behind On-Policy Delta Distillation, or OPD², described in a new arXiv paper by Byeongho Heo, Jaehui Hwang, Sangdoo Yun and Dongyoon Han.

The trick is what the authors call the delta signal. Rather than have the student chase the teacher's full output distribution, the reward is defined as the difference between the teacher model and its base model prior to instruction tuning for reasoning capability. In their framing, that difference captures the changes induced by reasoning tuning and provides a more direct signal for transferring reasoning capabilities. Everything else, the generic pretraining behavior the two models already share, drops out of the objective.

Why it matters if you are not writing distillation code yourself: on-policy distillation has become a common way to move capability from an expensive teacher into a cheaper student without leaning on a separate reward model. If the reasoning delta really is the load-bearing part, then training runs get shorter and the resulting student is better targeted at the skill you actually wanted. The authors report OPD² consistently outperforms conventional on-policy distillation across mathematics, science, and code-reasoning benchmarks, and enables reasoning LLMs to achieve strong performance with only a short post-training period.

The honest caveat is that the abstract does not give the specific benchmark scores, the exact teacher and student model families, or the compute-cost comparison, so take the framing as the authors' claim rather than an independently reproduced result. Code is promised at github.com/naver-ai/opd2, which is where the specifics will get pressure-tested.

For anyone shipping small reasoning models, the useful bet is that you no longer need both a giant teacher and a giant training budget, only the difference between what your teacher knew before and after reasoning training.