MRTPO assigns each reward to its own tokens, outperforms GDPO

After ten training steps, MRTPO produced the required tool-call format on 97% of BFCL cases. GDPO managed 2%. GRPO managed 4%.

That gap, from a new paper on Hugging Face by Haonan Wang (NUS), Zhining Liu (UIUC), Linxin Song (USC), Xuwei Ding (Penn), Jingcheng Liang (Minnesota) and Jian Kang (MBZUAI), is the sharpest demonstration of a plumbing fix for how reinforcement learning assigns credit when a language model is trained against several verifiable rewards at once.

The standard move today, GDPO, normalizes each reward separately so that distinct reward combinations keep distinct advantages. It then spreads every reward's advantage uniformly across every token of the response. The authors call out why that hurts: 'if a tool call specifies all arguments correctly but omits the required <think> tag, the format penalty is imposed equally on the correctly generated arguments.' A reward decided by a few positions receives most of its signal on tokens that have nothing to do with it.

MRTPO, which stands for Multi-Reward-to-Token Policy Optimization, replays each sampled response twice: once with the full verifier context appended to the prompt, once with one verifier's information block removed. The per-token Jensen-Shannon distance between the two next-token distributions is used as a relevance score, and each reward's advantage is reallocated toward its high-relevance tokens with mixing coefficient β=0.5. No auxiliary reward model, no token-level annotations.

In numbers, the paper reports that 'MRTPO has the best average performance on all three tasks and all six models.' On IFBench, whose constraints are not seen during training, the gain over GDPO is 2.4 and 3.2 points; on text-to-SQL, 1.3 and 1.6 points in average accuracy; on BFCL tool calling, the method mostly wins on speed. The median number of steps to reach a format reward of 0.8 is 13 for MRTPO, 19 for GDPO and 34 for GRPO. Text-to-SQL responses also become 40% shorter than GDPO's with Coder-3B, 'because reasoning that does not affect the query gets less execution advantage.'

What replay costs: because MRTPO only replays a response for rewards whose values vary within a group, 'overall, 12% to 21% of the response–reward pairs are replayed, and a training step takes 10% to 20% longer than with GDPO on the same GPUs.' Code is on GitHub. The paper sits in a visible run of recent post-training work tracked on AI Weekly, including Alibaba Cloud's Waypoint-OPD on gated RL distillation from earlier the same day.