Paper: Multi-Turn RL Lifts LLM Agent Recovery, Not Step Quality

Multi-turn reinforcement learning produces LLM agents that finish more tasks but pick worse individual actions. In a new arxiv paper hosted on Hugging Face, the authors show that RL-trained Qwen3 agents choose a better single-step action than the base model in only 42.8% of FrozenLake states and 46.8% of DeepCoder-MT states, with average step quality "no better than, even slightly below, the base model."

Where the RL model pulls ahead is error recovery. Started from identical error states on FrozenLake, the RL policy reaches success 51.3% of the time against the base model's 38.6%; on DeepCoder-MT the gap is 38.5% versus 22.9%. The paper frames this as the real mechanism of trajectory-level gains: multi-turn RL "primarily strengthens error recovery and long-horizon resilience," rather than uniformly improving step-level action quality.

That split has a deployment edge. The authors report that under tight interaction budgets, RL's advantage "diminishes and can even reverse on certain tasks," even when structural headroom remains. Their training-dynamics plot on DeepCoder-MT shows the trajectory-level reward rising by 0.331 while the per-step reward drops by 0.097 — gains and regressions moving in opposite directions over the same run.

Their fix, BudgetMix, treats the interaction budget as a training variable, sampling budgets at 10% intervals during RL. The paper reports average gains of 7.5 percentage points at 25% budget, 5.3pp at 50%, and 4.0pp at 75%, with parity at full budget. On BFCL, dense process-supervision methods like GiGPO and PPO "fail to enhance performance" under constrained budgets, so the budget-aware schedule is doing work that denser reward shaping did not.

The work lands in a crowded week for agent-RL writeups — this is the third agent-centric paper in our agents feed dated 5 November 2026, alongside the FrontierPhysics benchmark and a turn-level credit RL paper from Fudan and JD. Which model scales this result beyond Qwen3-4B and Qwen3-8B is not reported here.