Cliff Reward Shaping Beats GRPO 7%, Distillation 15% on LLM RL

Cliff, a reward-shaping method for RL-trained LLMs, splits every rollout at the point where reasoning first goes wrong and, per its authors, beats on-policy distillation by 15% and standard GRPO by 7% across 12 scenarios.

The paper, from Peixuan Han with co-authors including Gerald Friedland and Chris Kong, starts from a simple observation about verifiable-reward RL: 'once a reasoning process first goes wrong, evaluating the subsequent reasoning provides limited additional information, as it is already conditioned on an invalid prefix.' Cliff hands each rollout to an off-the-shelf LLM playing teacher, has that teacher locate the first mistake, then feeds the RL trainer token-level advantages, positive on the correct prefix and negative on the incorrect suffix.

The pitch is that this avoids the two crutches most process-reward setups lean on. There is no dedicated process reward model to train, and there is no assumption that teacher and student reason the same way. The authors write that Cliff 'consistently improves reasoning performance, outperforming on-policy distillation by 15% and standard GRPO by 7%, even with teachers of modest capability.'

The 15% and 7% numbers are aggregate across those 12 scenarios; the abstract does not break them out per benchmark or name the specific student model families or teacher LLM used. It lands in a busy stretch for fine-tuning research on AI Weekly, where yesterday's Meta and Princeton distillation paper made a related argument about how much intermediate signal is worth carrying through post-training.