arxiv.org web signal

Re:Max replay buffer counters RLVR solution mode collapse

By Published Updated

TL;DR

  • RLVR post-training concentrates probability onto fewer correct answer modes even as accuracy holds or improves, the paper reports.
  • Frontier models already show highly concentrated solution distributions before any further RLVR fine-tuning, according to ModeBench measurements.
  • The proposed Re:Max method keeps one verified example per mode in a replay buffer, improving both success and solution diversity.

Reinforcement learning with verifiable rewards concentrates language models onto fewer correct answer modes even as accuracy holds or improves. That is the finding of a new arXiv paper by Liv G. d'Aliberti, Marwa Abdulhai, Sofiia Druchyna, Peter Henderson, and Manoel Horta Ribeiro.

The authors built ModeBench, a benchmark whose verifier reports both whether a solution is correct and which mode of correct answer the model produced. Running RLVR against the benchmark, they report that "RLVR post-training concentrates probability onto fewer correct modes even as accuracy holds or improves," and that frontier models arrive already highly concentrated.

Their proposed fix, Re:Max, keeps a replay buffer that stores one verified example per discovered mode and trains uniformly across those stored modes. A solution the model has stumbled on once is practiced as often as one it keeps finding. The paper reports that "across three model scales, two RL objectives, and harder task constructions, replay improves both how often a policy succeeds and how many different ways it can succeed."

Two researchers from our Who's Who directory have already circulated the preprint.

Shared on Bluesky by 2 AI experts

Originally reported by arxiv.org

Read the original article →

Original headline: Measuring and Mitigating Solution Mode Collapse in RLVR