World Model RL trains 4B agent to beat 120B research rival
TL;DR
- Post-trained 4B and 9B research agents outperform 48B and 120B open-weight agents on held-out benchmarks, the paper reports.
- World Model RL replaces sandbox execution with a learned world model and accelerates training by 3-4x across tasks and agent scales.
- Two corrections, Online Debiasing and Inverse-Variance Denoising, are added to offset bias and suppress noise in the world model's rewards.
Trained with a learned world model standing in for the usual sandbox environment, 4B and 9B research agents outperform open-weight rivals of 48B and 120B on held-out benchmarks, according to a preprint on arXiv from Xiyuan Yang and collaborators. The method, World Model RL, delivers a 3-4x training speedup across tasks and scales.
The authors frame the problem as "a fundamental tension when scaling RL for these agents": the two halves of every trajectory scale differently. Generation batches across compute; execution "occupies its exclusive sandbox and real machine time." As trajectories grow longer, sandbox time dominates the bill.
WMRL routes around that by letting a learned world model stand in for the sandbox during training. Because the model is imperfect and its rewards are "corrupted by bias and noise," the paper adds two corrections, Online Debiasing and Inverse-Variance Denoising, which the authors say "strictly improve the convergence guarantee."
The abstract publishes no per-benchmark accuracy numbers, does not name the specific 48B and 120B open-weight agents used for comparison, and does not disclose what training the world model itself cost. The authors also report that WMRL transfers to post-training embodied vision-language-action policies, which they offer as evidence of generality.
Originally reported by paper
Read the original article →Original headline: World Model RL Lets 4B AutoResearch Agent Beat 120B Rival at 3–4× Lower Training Cost