Zhang survey maps on-policy distillation under one taxonomy

A new survey tries to do something the on-policy distillation literature has been quietly needing, which is to put the various losses and tricks under one taxonomy you can actually navigate. The paper, posted to arXiv by Bowen Zhang, frames on-policy distillation as "a feedback-to-update problem rather than a single loss family", with two routes feeding the update: direct distributional losses, and policy-gradient-style log-ratio updates.

The plain-language version, which the paper itself states, is that the student model generates rollouts, a teacher or self-teacher scores those tokens in the contexts the student actually produced, and dense log-probability or distributional signals get turned into training updates. That "states induced by the current or recent student policy" detail is the part that makes OPD different from vanilla distillation on a fixed dataset.

What's useful in the survey is the separation of two mechanisms that often get tangled in stability discussions. Temporal credit is how teacher-student log-ratio returns weight sampled actions across rollouts. Vocabulary routing is where probability mass actually moves when negative feedback suppresses a token. Conflating those, the paper argues, is part of why OPD failure modes feel mysterious. The author lists effectiveness factors as "state compatibility, support construction, temporal credit, vocabulary-level probability routing, gates and weights, and regularization", which is a longer checklist than the usual KL-direction debate.

On top of the taxonomy, two new directions are proposed. GAE-OPD is offered as a value-based hypothesis for the log-ratio returns. Counterfactual Routed OPD (CR-OPD) tries to push probability mass toward "teacher-supported, student-reachable alternatives" rather than just down on the bad token. Take both as research agenda items, not validated results, since this is a survey rather than an empirical study.

The honest caveat is that what the paper does not give you is benchmarks. There are no win-rate tables, no compute budgets, no head-to-head comparisons between methods on a shared task. If you are picking a distillation recipe this quarter, the survey is a map, not a verdict. The forward-looking part is whether a lab with the compute to actually run GAE-OPD or CR-OPD picks up the framing and reports results. That is the moment this becomes more than a taxonomy.