NGU sampling method targets RL's 'Matthew Effect' in LLMs

Reinforcement learning makes language models much better at problems they were already close to solving. On the hard ones, the gains stay small.

That is the finding of a paper on arXiv by Michael Noukhovitch, Hamish Ivison, Nathan Lambert and Aaron Courville. "RL shows large improvements on easy problems that an LLM is already good at solving, but small improvements on hard problems," the authors write. They call the pattern "the Matthew Effect in RL for LLMs," borrowing the "rich get richer" phrase from economics and network science.

Their argument is that current RL pipelines are making the imbalance worse by "wasting too much compute on easy problems." The proposed fix is Never Give Up, or NGU: a sampling loop that "keeps generating samples for a problem until one is correct," run on top of asynchronous RL so that easy items get filtered out cheaply and the freed compute is redirected to the harder ones.

On the math benchmark Deepscaler, the authors report that "NGU improves performance per compute, especially on harder problems." On Manufactoria, a coding task, they say standard GRPO with a per-test reward fails to fully solve problems mixing easy and difficult tests; NGU "iteratively improves, solving harder and harder tests, until it learns to fully solve coding problems." The paper also discusses off-policy robustness as one of the design choices that affect NGU.

The abstract publishes no absolute scores, no compute figures, and no comparison numbers against a fixed-sample GRPO baseline. Two of the researchers we track had already circulated the link by the time we picked it up.