SERO Teaches Agent RL via Executable Oracle Continuations

Long-horizon agentic RL has a chicken-and-egg problem: if a policy's rollouts never reach a useful state, improved credit assignment can't supervise what was never encountered. State-Rooted Executable Rollout Optimization — SERO — tries to break that loop by letting a training oracle finish what the policy started, according to the preprint on arXiv.

The method restores "a state actually reached by the current policy" and lets a training-only oracle continue interacting with the real environment. The authors write that this makes "the downstream observations and oracle actions themselves available for learning." A second ingredient, policy-reachability weighting, scales each oracle step by how compatible the frozen behavior policy already is with the preceding oracle actions, so "deeper supervision" turns on as the policy catches up. No oracle is used at inference.

On ALFWorld and WebShop with Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct, "learning the full executable continuation is substantially more effective than supervising only the root oracle action," the paper reports, with the reachability weight "further" improving out-of-distribution performance. The abstract gives no absolute score numbers.