ReOPD Cuts Agent Distillation Rollout Cost 4x via Prefix Replay

A new arXiv preprint from Baohao Liao, Hanze Dong, Christof Monz, Xinxing Xu, Li Dong and Furu Wei is worth flagging for anyone staring at an agent fine-tuning bill. The paper, "Multi-Turn On-Policy Distillation with Prefix Replay", argues that most of the cost of on-policy distillation for tool-using agents comes from the environment loop itself, not the gradient math, and it proposes a way to keep the on-policy training signal while cutting that loop out during student training.

The setup is that fully online on-policy distillation, or OPD, teaches a student model by having it act in an environment turn after turn while a teacher supervises. That is expensive because, as the authors put it, "each update requires fresh student rollouts through the environment." Their alternative, Replayed-Prefix On-Policy Distillation or ReOPD, reuses pre-collected teacher trajectories instead. The paper reports it is "at least 4× faster per rollout than OPD" and "uses zero tool calls during student training," tested on mathematical reasoning with Python and search environments across multiple model scales.

The mechanism worth understanding is what the authors call a "two-sided distribution shift between student occupancy and teacher reliability." Aligning the replayed histories with what the student would actually do makes the data more relevant but less trustworthy, because the teacher's supervision was recorded under different conditions. Their answer is a step-decaying sampling schedule that leans on early, lower-shift prefixes and de-emphasizes later, drifted ones.

Why this matters if you are budgeting agentic post-training: environment interaction has been the line item that scales worst, because every gradient step drags along tool calls, latency and, for hosted tools, per-call costs. A 4× reduction in rollout cost with reportedly preserved accuracy shifts the economics from a per-run environment bill toward a fixed dataset cost. The honest caveat is that the evidence is on math, Python and search environments, and the abstract does not spell out absolute accuracy numbers against a fully online OPD baseline, so take "preserves or improves" as the authors' framing until independent replications land. What the write-up also does not give you is how often the frozen prefix set has to be refreshed as the student improves, or how the schedule holds up on messier tool ecosystems.

If the pattern generalizes, the interesting downstream effect is that trajectory datasets become the reusable asset, and smaller groups without continuous environment budgets get a clearer path into on-policy agent distillation.