CLSA reuses routing index across layers for 7.6x decode speedup

A new preprint claims up to 7.6x decoding speedup and 17.1x overall throughput at 128K context by changing one thing about sparse attention: computing the top-k routing index once and reusing it across decoder layers. The approach, called cross-layer sparse attention or CLSA, builds on YOCO-style KV-sharing architectures and extends the sharing from the cache to the router itself. The paper is on arXiv.

The abstract frames the trade-off bluntly. "Structured block sparse methods typically provide stronger acceleration but incur noticeable quality loss, while token sparse methods are usually more accurate yet deliver limited end-to-end speedup because top-k routing over the full cache remains expensive." CLSA's pitch is to keep the token-level selectivity and amortize the cost: "A single indexer computes token-level top-k selection once and reuses the resulting index across layers, thereby preserving the fine-grained selectivity of token sparse attention while amortizing the routing overhead."

The authors say CLSA improves "pre-filling, KV-cache storage, and long-context decoding" together, the three bottlenecks a long-context decoder has to hit at once.

The abstract names no specific base model, no hardware, and no per-benchmark numbers. Readers get the headline speedups, the architectural claim, and a wait on the full paper. The link was moving through a couple of the researcher accounts we track within hours of posting.