SpecPilot Lifts Speculative-Decoding Speedup to 10.87× From 4.46×
TL;DR
- SpecPilot reports raising speculative-decoding speedup over autoregressive from stock's 4.46× to 10.87× with one request, and from 1.94× to 3.93× with batching.
- The agentic loop discovered a general lossless 12-bit weight format, tree verification for hybrid recurrent targets, two-level verification and three batching mechanisms.
- Authors from UC Berkeley, Princeton, Stanford and NVIDIA evaluated across four target models, three drafters, two GPU platforms and five benchmarks, without training or quality loss.
SpecPilot, an autonomous agentic loop from researchers at UC Berkeley, Princeton, Stanford and NVIDIA, reports raising the speedup of speculative decoding over autoregressive decoding "from stock's 4.46× to 10.87× with one request, and from 1.94× to 3.93× with batching," without training or quality loss.
The system treats speculative-decoding tuning as a search problem. "In each round, a cost model locates the largest gaps between the current system and its ceiling," the abstract says, and parallel agents explore directions and integrate the successful ones. Token forcing "checks every output token against stock to detect reward hacking," and a structured memory carries validated mechanisms forward to later rounds and deployments.
The loop's output, per the paper: "a general lossless 12-bit weight format, tree verification for hybrid recurrent targets, two-level verification and three batching mechanisms." The authors evaluate across four target models, three drafters, two GPU platforms and five benchmarks, none of which the abstract names.
It joins a busy stretch of inference-infrastructure research on our tracker, where sub-linear gains in drafter acceptance or verification overhead translate directly into cheaper tokens served.
Originally reported by arxiv.org
Read the original article →Original headline: Berkeley, Princeton, Stanford and Nvidia's SpecPilot Autonomously Discovers New Speculative-Decoding Algorithms