TurnOPD trims wasted turns in long-horizon agent distillation

A new paper on arXiv from Yuhang Zhou, Kai Zheng, Haoling Li, Dengyun Peng, Can Xu and Jingjing Chen makes a small but pointed claim about how we train language agents: in on-policy distillation for long-horizon tasks, full-horizon rollouts waste wall-clock resources on tail turns that carry weak and noisy KL supervision. Their method, TurnOPD, is a turn-aware rework of vanilla OPD that tries to spend compute where the learning signal actually lives.

The fix has two moving parts. The first is adaptive rollout-depth budgeting, which uses probe-based turn statistics to decide how long each rollout should actually run rather than always going to the horizon. The second is a progressive turn-normalized loss that gradually shifts the KL weighting from token-level to turn-balanced supervision, so training does not keep piling gradient on shallow, easy tokens while under-training the deeper decision turns. Neither piece requires an architecture change, which is part of the pitch.

Why this matters if you are not writing distillation papers: on-policy distillation is one of the main ways smaller open models are being taught to behave like larger agentic ones, and long-horizon tasks are exactly the place the bill runs up. If turn-aware budgeting really does move the accuracy-efficiency frontier, it is a lever any team training agents on top of a bigger teacher can pull without changing the stack. The authors report evaluations on ALFWorld, WebShop and Multi-Hop Search, framing the result as superior validation accuracy under equal wall-clock training budgets against vanilla OPD.

The honest caveat is that the abstract does not disclose the specific numbers, so take the framing as a directional claim rather than a settled magnitude, and the three benchmarks are the usual agentic ones rather than open-ended web or coding agents where turn distributions look quite different. What the paper does not give you is a read on which teacher and student model families were used, or how the probe-based turn statistics behave when trajectory lengths are highly variable.

Still, the direction is the interesting part. Once you accept that not all turns of an agent trajectory are worth the same gradient, the design space for cheaper, better agent training opens up in a way that benefits the smaller, cost-sensitive labs most.