CoRT adds token-level credit to GRPO for a 4.4-point gain

For anyone actually training instruction-following models with rubric rewards, a new paper from Bo-Wen Zhang, Junwei He and coauthors is worth a read, because it goes after a specific inefficiency in how GRPO consumes structured feedback. Rubrics decompose a response into criteria like correctness, formatting, or safety, but by the time GRPO uses them, everything is collapsed into a single scalar reward and broadcast uniformly to every token. Formatting tokens and content tokens get the same credit even when only one of them was actually responsible for satisfying a criterion.

CoRT, short for Counterfactual Replay for Token-Level Rubric-Guided Policy Optimization, takes a different route than training a separate token-level scorer. According to the paper on Hugging Face, it rescores the same sampled response twice: once with the rubric criteria in the prompt, once without them. The tokenwise log-likelihood contrast is used as a proxy for how much each token depends on the rubric, and CoRT maps those contrasts to bounded, response-normalized weights that redistribute the signed GRPO advantage across tokens. No auxiliary scorer, no change to the response-level reward, no separate relevance-learning stage.

The headline result is an average gain of 4.4 percentage points over matched response-level GRPO across instruction-tuned models and reward granularities. The main experiment matrix trains Qwen3-4B-Instruct and Qwen2.5-7B-Instruct under both Constraint Satisfaction Rate and All-or-Nothing rewards, and validates on IFBench, IFEval, MultiDimIF, and AdvancedIF, the last of which uses o3-mini as its judge. Extension runs on Qwen3-14B show the effect largely holds at that scale, and plugging CoRT into DAPO and GSPO produces further improvements on most reported metrics, which suggests the credit-allocation trick is fairly orthogonal to the underlying policy objective.

The honest caveats sit in the paper itself. Under AON on Qwen3-14B, CoRT trails GRPO on IFEval even as it wins on IFBench and MultiDimIF, so gains are not monotonic across metrics. The authors also note the replay signal weakens when criteria barely affect token likelihoods, are redundant with the prompt, or when the response-level reward is noisy. What the reporting does not give you is any evidence outside the Qwen family, outside instruction following, or past the 500-step training window they use for validation.

For teams shipping rubric-graded RL pipelines, the appeal is that CoRT is a drop-in modification to an existing GRPO stack rather than a new scoring model to train and maintain. If the result reproduces on other model families, the practical win is fewer moving parts for the same or better instruction-following accuracy.