HyCo cuts server GPU time 38.6% via mobile-cloud drafting

HyCo, a new speculative-decoding preprint, splits the draft tree between a mobile NPU and the cloud: deeper branches run through the server model's native multi-token prediction head executed on the phone, while alternative branches use one pass of a small language model. Instead of running sequential draft steps, the server "returns the hidden state, input embedding, and key-value entries needed by the head."

The paper reports server GPU-time cuts per output token of 38.6% on Spec-Bench and 29.2% on 32K-token summarization versus server-side MTP self-speculation, tested on a Galaxy S26 Ultra over LTE/5G. Against an SLM-only drafter, three Spec-Bench runs show "9.0 percentage points more saving and 1.9x delivery speed," with quality within +0.2 judge-score points of a Cloud+MTP baseline (paired 95% CI: -0.9 to +1.3). Those headlines rely on an ε-threshold rule that relaxes exact preservation of the verifier's sampling distribution; without it, savings drop to 27.7% and 11.9% on the two datasets. Both drafters are quantized, use a restricted vocabulary, and sit inside a 1K-token window.

Two researchers we track flagged the paper.