Google's Muir cuts perplexity 3.6% vs Transformer at 1.34B

Google researchers Gil Tabak and Clemens JS Schaefer have posted a linear-attention paper claiming their architecture, Muir, reaches 1.8–2.0% lower perplexity than Gated DeltaNet-2 and 2.5–3.6% lower than a Transformer baseline across models from 166M to 1.34B parameters.

The design starts from Gated DeltaNet-2 and adds two theory-driven changes. State updates, the authors argue, are "far less sensitive to erasure than to writes," so the erase gate can be compressed harder than the write gate. Fusing all five gates into one "sensitivity-allocated low-rank projection" cuts gate parameters from 9.44M to 2.10M at d=2,048; the paper reallocates that budget into a 30.4% wider FFN. A "matched-filter readout gate" then re-weights state rows at query time.

The long-context numbers are the showiest claim. Trained at a 4K sequence length, the 1.34B Muir extrapolates to 10M-token streams "without perplexity drift," a 2,560× stretch, and the paper reports "near-100% passkey retrieval to 1M tokens" from the static readout — while GDN-2 collapses at 4K–8K. At that horizon Muir's recurrent state is 25 MB in float32, against what the paper describes as a 2.1 TB bfloat16 KV cache for a Dynamic YaRN Transformer. Training throughput matches GDN-2 and runs 2.7× faster than the Transformer at 32K context on 8 TPU7x chips.

The gains outside long-context stress tests are smaller: up to 1.9 points on commonsense reasoning. All figures are the authors' own, and the paper lands in a crowded field — our tracker has logged 339 generative-AI stories in the last 90 days, with architecture papers a steady part of that flow. Google has not said whether Muir will ship inside a product.