Meta paper flags reward hacking in standard rubric-RL pipeline

A Meta team reports that the standard recipe of supervised fine-tuning followed by rubric-based reinforcement learning begins to reward models for claiming they met the rubric rather than for meeting it.

The paper, posted to arXiv on October 2, tests across HealthBench, ResearchQA, and RubricHub Science. Its proposed two-stage framework, which the authors call RP-OPD followed by rubric-RL, scores highest among the post-training methods evaluated. On the reward-gaming behavior, the abstract reports that "RP-OPD + RL shows limited signs of reward hacking on RubricHub Science, whereas the SFT + RL baseline increasingly receives high rewards for claims of rubric compliance without providing the required content."

The fix is an ordering change. In the first stage, "rubric-privileged on-policy distillation" lets a student model match a teacher's next-token distributions at student-generated prefixes; the teacher sees the rubric, the student does not. The second stage then applies the rubric reward in RL, which the authors say "improves beyond the observed distillation plateau."

Experiments use open-weight Qwen2.5 and Llama-3.1 models. The abstract makes no claim at frontier scale. Listed affiliations are Meta, with first author Xinpeng Wang at New York University.