AgentOPSD scores turn-level credit for long-horizon agent RL

Credit assignment is the boring, load-bearing problem in agentic reinforcement learning. Give a model twenty turns to finish a task, hand it a single reward at the end for success or failure, and it has almost no principled way to know which of those turns actually mattered. Trajectory-level advantage estimates smear credit across everything the agent did, and critic-based fixes come with their own instability. A new arxiv preprint proposes AgentOPSD, a critic-free method that tries to extract turn-level credit from sparse outcome rewards without adding extra rollouts.

The mechanism is where it earns attention. The authors aggregate token-level teacher-student log-probability gaps into a per-turn evidence signal, then recursively update a Bayesian belief state in log-odds space, so each turn's credit reflects how the belief shifts between consecutive states. That gives them two things at once: a reweighting scheme that converts a single end-of-trajectory reward into denser per-turn supervision, and an implicit pivotal-turn readout via the marginal belief revision. Both slot into standard policy optimization, and neither needs an additional critic or extra rollouts.

Results are reported on three of the usual agent benchmarks, ALFWorld, WebShop and Search-QA, using Qwen2.5 at 3B and 7B. AgentOPSD outperforms GRPO and strong self-distillation baselines, reaching 89.1% success on ALFWorld with the 7B model. Ablation studies attribute the gains to two ingredients specifically: aggregating credit at the turn level rather than the token, and letting the belief update carry history between turns.

The honest caveat is what the paper does not give you. ALFWorld is a scripted embodied benchmark and WebShop and Search-QA are constrained tool-use environments, so how far this transfers to open-web browsing or production agent stacks is untested here. The method also assumes a privileged teacher is available for distillation, so teams without a stronger model on hand do not get the setup for free. The reporting also does not lay out compute overhead of the recursive belief tracking against a plain GRPO run.

For teams shipping agent workflows on open 3-7B models, the appeal is straightforward: a drop-in credit-assignment scheme that plays with the optimizer they already use, and an interpretable pivot-turn signal that value-function critics do not surface. If it holds outside the benchmark set, it is the kind of primitive that quietly becomes standard.