Paper: Linear map reuses KV cache across LLM sizes, 2.7-25x faster

There is a quiet cost in serving multiple sizes of the same model family: every time you swap between, say, a 14B and a 32B sibling, you pay to prefill the prompt again on the new model. A new arxiv paper from a group including Taekyung Heo, Rasoul Shafipour and Bita Darvish Rouhani argues you may not have to, because the key-value cache from one model in a family carries enough linear structure to be mapped onto another.

The core observation is that KV states across matched pairs of models are more predictable from each other than you might expect. In a Qwen3 14B to 32B transition, the authors report that a single source layer explains 56% of key variance and 32% of value variance in the target. That is the opening for a closed-form fix: a three-step ridge regression mapper that picks the top-k predictive source layers per target layer, strips RoPE from the keys so the mapping is position independent, and calibrates on 500 FineWeb-Edu sequences of 1,024 tokens each.

The reported numbers are the interesting part. Across six model pairs spanning three families, the linear mapper retained 73-98% of task accuracy on four pairs and ran 2.7 to 25 times faster than a full re-prefill. The other two pairs degraded significantly, which the authors patch with a nonlinear MLP variant that recovers up to 37 percentage points of HellaSwag retention on the failure cases. Stability across multi-turn conversations is also claimed.

The honest caveat is that a third of the tested pairs failed the linear approach outright, and the paper's calibration set is short, English, web-educational text. What the report does not tell you is which specific pairs broke, whether fine-tuned checkpoints in the same family still map cleanly, or how the mapping behaves on the long-context, tool-heavy traffic that actually dominates real inference bills.

Still, the direction is worth watching. If a mapping this cheap holds up on the traffic that serving providers actually see, tiered routing between a small and a large sibling stops being an expensive design choice and starts looking like a default.