Marek, Ryabinin: score centering stabilizes LLM RL training
TL;DR
- Marek and Ryabinin argue RL instability under training-inference mismatch is primarily caused by drift, a bias that accumulates every training step.
- Their additive score-centering term alone matches or outperforms importance-sampling methods under quantization, with the gap widening as mismatch worsens.
- Because the correction is additive, it composes with importance sampling and beats pure IS baselines in staleness experiments at up to 30B parameters.
Reinforcement learning on large language models breaks when the training engine and the inference engine disagree, and in a new arxiv paper Martin Marek and Max Ryabinin argue the breakage has one dominant cause: drift, which they define as "a persistent bias between training and inference engines that accumulates with every training step."
Their fix is a single additive correction term they call score centering. In experiments across models from 0.6B to 30B parameters, the authors write that "score centering alone matches or outperforms methods based on importance sampling under quantization, with the gap growing as the mismatch becomes more severe." Because the correction is additive, it also composes with importance sampling. Their combined method, the paper reports, "outperforms pure importance-sampling baselines in our staleness experiments."
The abstract names no specific benchmarks, no training recipe, and no absolute numbers behind the "matches or outperforms" phrasing. It also stops at 30B parameters, well below the scale of today's post-trained frontier systems.
Shared on Bluesky by 1 AI expert
Originally reported by arxiv.org
Read the original article →Original headline: Score Centering Stabilizes Off-policy Reinforcement Learning