MPO Paper Shifts Agent RL Credit From Token to Turn Scale
TL;DR
- The authors argue critic values in long-horizon LLM agents move mainly across turns, not tokens, so credit should be anchored at the turn scale.
- MPO pairs Multiscale Advantage Estimation with turn-level importance sampling to align both intra-turn token credit and policy updates with turn boundaries.
- On search, tool-use, and automation tasks at 4B and 30B-A3B, the authors report consistent gains over strong RL baselines.
A 19-author group led by Tao Ren is arguing that the usual way of training long-horizon LLM agents with reinforcement learning is pointed at the wrong unit of credit. An agent emits actions token by token but talks to its environment one full turn at a time, and the critic's value signal moves mostly across those turns, not across the tokens inside them.
The proposed fix, which the authors call Multiscale Policy Optimization, splits credit assignment across two scales. "Critic values vary primarily across turns, motivating optimization anchored at turns with finer credit allocation among tokens," the paper states. In practice that becomes two moving parts: Multiscale Advantage Estimation for intra-turn token-level credit, and turn-level importance sampling so that the policy update itself is keyed to the turn boundary.
Across search, tool-use, and automation tasks, at both 4B and 30B-A3B parameter scales, the authors report "consistent gains over strong RL baselines." The abstract puts no per-task numbers on the record and names no specific baseline methods; the paper went up on October 26 and is single-group work with no outside replication in hand yet.
Originally reported by paper
Read the original article →Original headline: Multiscale Policy Optimization Fixes Long-Horizon Agent Credit Assignment, Earns 375 HF Upvotes in 72 Hours