Specter MoE System Reports 1.88x Peak Single-GPU Speedup
TL;DR
- Specter reports an average 1.50x speedup in time per output token, peaking at 1.88x, against the state-of-the-art MoE inference system on single-GPU hardware.
- A quantized draft model runs on the GPU while experts live in host memory; multi-step routing lookahead lets the runtime pre-stage expert transfers over PCIe.
- The authors have released code and the measured gains cover single-request inference only, not batched or multi-tenant serving.
Specter, a new MoE inference system, reports an average 1.50x speedup in time per output token with peaks hitting 1.88x over the state-of-the-art on commodity single-GPU hardware, according to its authors on Hugging Face. The paper targets a familiar bottleneck. Mixture-of-experts models often exceed a single accelerator's memory, so experts get pushed out to host memory, and the resulting PCIe traffic sits on the critical path because routing is data-dependent.
The method pairs quantization with speculative decoding. "Its lightweight quantized draft model runs efficiently on device while preserving high-fidelity expert activation patterns," the authors write, which gives the runtime "accurate multi-step lookahead over future expert usage" so expert movement can be organized ahead of verification rather than reactively.
Code is posted at the paper's GitHub repository. The reported speedups cover single-request inference; the abstract does not break out batched-throughput or multi-tenant numbers. The result arrives in a busy week for MoE-inference work on our tracker, alongside Lafa's flow language model earlier the same day.
Originally reported by huggingface.co
Read the original article →Original headline: Specter Pairs Quantized Speculation With Expert Offloading, Hits 1.88x MoE Inference Speedup on Single GPUs