Multi-Branch Transformer Cuts KV Cache Up to 34%, Boosts Throughput Up to 51.4%

Found first: a primary source the press has not covered yet.

Researchers at the University of Illinois and Cornell have published a transformer architecture that runs blocks in parallel rather than purely in sequence, cutting KV-cache storage by up to 34% and lifting decoding throughput by up to 51.4% on a single H200 GPU. The paper, Multi-Branch Transformer (arXiv:2611.19800), was submitted on November 15, 2026.

What the source says

The Multi-Branch Transformer (MBT) passes input through shared initial layers, then routes it through independently parameterized branches of sequential blocks running in parallel, before fusing into shared final layers. Three variants are defined: Wide-Shallow, Narrow-Deep, and Balanced. A Joint-KV mechanism shares KV-cache construction across branches, which is the direct source of the inference gains. Scaling experiments run from 0.8B to 2.1B total parameters; the 2-branch Wide-Shallow variant consistently outperforms sequential baselines at matched compute budgets. Standard transformers require up to 1.26x as much training compute to match MBT's validation loss.

Why it matters

Transformer architecture research has pulled two levers: width and depth. MBT proposes parallel branching as a third, and backs it with scaling laws across four model sizes and H200 hardware benchmarks at a realistic serving configuration of 16K input tokens and 64 concurrent requests. The 34% KV-cache reduction translates directly to either lower memory cost or longer context at the same hardware budget. The compute-efficiency result means a given quality threshold costs less training compute than a standard transformer, which is immediately relevant to labs running continuous pretraining runs.