Prefix Sliding paper reports 3x reasoning speedup, no retraining
TL;DR
- Prefix Sliding claims a 3x inference speedup on reasoning models with no retraining, by discarding tokens outside the prefix and a recent-context window.
- Training the method with reinforcement learning enables reasoning traces beyond a hundred thousand tokens with better performance.
- Ablations report the approach outperforms both token summarization and vanilla sliding window baselines.
A new paper posted to arxiv by Niklas Muennighoff and a large group of coauthors claims a 3x inference speedup for reasoning models with no retraining, and reasoning traces "beyond a hundred thousand tokens" when the method is trained in with reinforcement learning.
The method, Prefix Sliding, throws away most of the middle of a long chain of thought. It keeps the prompt prefix, which the authors describe as "key instructions and tools available to the model," plus "the window of the last few thousand tokens." Everything in between gets dropped, capping total memory regardless of how long the model reasons.
The abstract's core observation is short: "we find most intermediate reasoning tokens lose importance as the model continues reasoning. This calls into question whether retaining them is worth the cost." Ablations, the authors write, show the approach "outperforms summarizing intermediate tokens or vanilla sliding window." Code is on GitHub.
The abstract names no benchmark numbers and no specific base models. Two of the researchers in our Who's Who directory shared the link.
Shared on Bluesky by 2 AI experts
Originally reported by arxiv.org
Read the original article →Original headline: Prefix Sliding for efficient test-time scaling