Found first: a primary source the press has not covered yet.
Researchers at the University of Illinois and Cornell University have proposed a transformer architecture that runs blocks in parallel rather than purely in sequence, reducing full-model KV-cache storage by up to 34% and lifting decoding throughput by up to 51.4% on a single H200 GPU. The paper, Multi-Branch Transformer, was submitted to arXiv on November 15, 2026.
What the source says
The Multi-Branch Transformer (MBT), from Gaotang Li, Lingjie Chen, Chunyuan Deng, Zhiwei Xu, Hongzheng Chen, Christopher De Sa, Jingrui He, Yingheng Wang, and Hanghang Tong, routes input through shared initial layers, then splits into independently parameterized branches of sequential transformer blocks running in parallel, and fuses before shared final layers. Three variants are studied across 0.8B to 2.1B parameter scales: Wide-Shallow, Narrow-Deep, and Balanced. A Joint-KV mechanism shares key-value representations across branches at each depth, which drives the cache reduction. The 2-branch Wide-Shallow variant is the strongest performer, requiring a standard transformer to use up to 1.26x as much training compute to match its validation loss. The throughput and cache figures are measured on a single H200 GPU at 16K input tokens with 64 concurrent requests.
Why it matters
Width and depth are the two canonical levers for scaling transformers; this paper proposes branching as a distinct third, with its own scaling laws derived across a ladder of model sizes. The inference gains are hardware-grounded: a throughput improvement of up to 51.4% at 16K context and 64 requests is a figure engineers can benchmark against directly. The Joint-KV design delivers the cache savings without a separate compression step, keeping the method straightforward to replicate. Labs running large inference fleets at long context lengths are the most immediate audience.