NTU's DeltaTTG Beats GRPO by 7.2 Points on Llama-3.1-8B Agents

A new paper from Nanyang Technological University and Renmin University of China proposes using the number of tokens an agent still has to generate before finishing a task as a cheap stand-in for how much work remains. The signal is called Token-to-Go, and the method built around it, DeltaTTG, reports an average 7.2-point gain over GRPO when training Llama-3.1-8B-Instruct across three agent benchmarks, and 5.7 points with Qwen3-4B.

The benchmarks are ALFWorld, PaperSearchQA and BFCLv3, covering interactive decision making, information seeking and tool use. A separate frozen 'progress meter' is first trained on recorded trajectories from diverse policies, then used during actor training to redistribute outcome credit across steps.

The authors describe the mechanism bluntly: 'During actor training, changes in the frozen meter's estimates redistribute outcome-derived advantages across steps without altering the reward, preserving each trajectory's advantage sign and token-average scale.' In other words, the reward model is untouched; what moves is how a trajectory's final advantage gets sliced across its individual actions.

Why use remaining tokens at all? The paper argues that 'beneficial actions from the same state leave fewer tokens to completion on average across many trajectories than harmful ones,' and that 'TTG labels come at no extra cost from recorded trajectories.' That is the pitch — a dense, per-step learning signal without external judges, human annotations or task-specific heuristics.

One concrete constraint sits inside the method: a dataset-dependent failure cost constant, which the authors cap with a token-multiplier of M=5 to 'help mitigate sensitivity to C and stabilize actor updates.' The abstract reports gains 'on average' across baselines and does not break down per-benchmark deltas or variance across seeds in the summary text. Code is published at the authors' GitHub.

The work lands in a dense stretch of credit-assignment research on our radar, including LinkedIn's distillation-based Δ-MOPD and today's OpenRubrics ReVeR rubric reward model, both circulating in the same multi-turn agent and fine-tuning threads.