OPD Paper: One Query Reaches 71.5% of Full-Data State Coverage

A new arXiv preprint by Zixuan Fu, Bingxiang He and eleven co-authors argues that on-policy distillation of large language models can be pushed to most of its full-data performance using a single training query, and that adding more data is not what has been driving recent gains. The paper, "Rethinking On-Policy Distillation of Large Language Models II: One Training Example", reports that one query reaches 71.5% state coverage, most of it inside the first 100 steps.

The authors measure state coverage as the fraction of states visited by full-data OPD that a smaller query set's rollouts also reach. Scaling up the query pool closes the rest of the gap: "Adding semantically distinct queries raises coverage and validation accuracy together, until 16 queries reach 98.9% and match full-data training." Yet the rate at which the student actually absorbs the teacher's supervision barely changes with dataset size. As the abstract puts it, "alignment slows at a similar pace whether OPD trains on one query or the whole dataset, and even a fixed set of states takes hundreds of steps to absorb."

From that, the authors draw a blunt diagnosis: "OPD is therefore data-overfed but algorithm-starved." Rollouts expose broad supervision quickly; the student ingests it slowly. The finding extends to multi-teacher OPD, where 16 semantically diverse queries per domain reportedly match full-data MOPD, and holds up when the queries are swapped for "content-light templates and off-domain WildChat queries," both of which approach the real-query baseline.

The abstract does not publish absolute benchmark scores, teacher-student pairings, or ablations across model sizes; the numbers cited here are the ones the authors put in the abstract itself of a 29-page preprint submitted this month. The paper is part of a visible run of post-training work on our radar, including a KL-distillation study from Meta and Princeton and a cliff reward shaping result from earlier in the week, part of 35 fine-tuning stories we have logged over the last 90 days.

The authors close by asking the field to redirect attention: they hope the findings "direct future work toward the step efficiency of OPD, and prompt a re-examination of the data and the mechanisms behind its recent successes in frontier post-training."