Kimi Delta Attention + Muon Tops Linear-Attention Study

Linear attention has been the quiet architecture-of-choice story of the last year, as teams looking to cut long-context inference cost cast around for something cheaper than softmax attention. The trouble was, nobody had put the competing variants on the same bench. That is what a new paper on arXiv by Tommaso Cerruti and colleagues sets out to do.

The team compare four recurrent linear-attention mechanisms, DeltaNet, Gated DeltaNet, Kimi Delta Attention, and Gated DeltaNet-2, against standard softmax attention, primarily at 350 million parameters trained on 15 billion tokens, with additional runs at 1.3B and 3B. The headline finding cuts against the easy assumption that DeltaNet has already won: Kimi Delta Attention paired with the Muon optimizer reaches the lowest final validation loss in the study. Muon itself, the authors report, consistently lowers final validation loss relative to AdamW across matched settings, which is a quiet but useful data point for teams whose optimizer choice has calcified. Throughput went the other way though, a pure Gated DeltaNet stack trained with AdamW clocked the highest normalized training throughput.

The second thing the paper does is propose a lightweight primitive called Cross-Layer Value Routing, or CLVR. It reportedly lowers final validation loss for both DeltaNet and Gated DeltaNet, which makes it the kind of drop-in module architecture teams can actually try on an existing stack this quarter without a rewrite.

The honest caveat is that the primary results sit at 350M parameters. That is a legitimate research scale, but production long-context serving typically lives one or two orders of magnitude bigger, and the write-up is lighter on the 1.3B and 3B runs. What the paper also does not give you is downstream task quality on real benchmarks, only validation loss, and no serving-hardware throughput numbers on the winning combinations. Both are things a platform team would want before standardizing.

Still, if you were about to choose an attention variant for a long-context inference stack, this is the first study I would send round the team. The gap it closes, an apples-to-apples ranking of the DeltaNet family plus an optimizer axis, is exactly the gap that has been costing teams weeks of ad hoc reproduction.