Soft-MTP: one-line fix outperforms standard MTP from 167M to 1.2B
TL;DR
- A new paper argues standard Multi-Token Prediction gradients contain a low signal-to-noise residual that interferes with learning and worsens with scale.
- Soft-MTP is a one-line change using a KL gradient-recovery trick to strip the residual, adding no parameters and no measurable overhead.
- Tested from 167M to 1.2B parameters, Soft-MTP shows lower NTP perplexity, faster speculative decoding and higher zero-shot accuracy at every scale.
A new arxiv preprint by Jayden Teoh argues that standard Multi-Token Prediction training has been quietly throwing away gradient signal, and the problem grows with model size.
The paper decomposes the MTP cross-entropy gradient into two pieces: a KL self-distillation term that pulls each MTP head toward the model's own next-token prediction distribution, and 'a residual that has a low signal-to-noise ratio and buries the useful KL signal.' That residual, the paper reports, 'interferes with both next-token and multi-token learning, increasingly so with scale.'
The fix, Soft-MTP, is a one-line code change that reuses the existing NTP and MTP cross-entropy computations to backpropagate only the KL signal. It supports both DeepSeek- and Meta-style MTP, adds no parameters, and 'incurs no measurable overhead.' From 167M to 1.2B parameters, the paper reports lower NTP perplexity, faster speculative decoding and higher zero-shot accuracy at every scale, with lower MTP perplexity from 689M onward and stronger math and code reasoning after fine-tuning at 1.2B.
Teoh calls it 'in effect, a free lunch for MTP optimization.' The experiments stop at 1.2B parameters, well below the scales where DeepSeek and Meta actually train.
Originally reported by paper
Read the original article →Original headline: One-Line Fix Removes a Hidden MTP Gradient Flaw That Gets Worse as Models Scale