Found first: a primary source the press has not covered yet.
Researchers at MIT, UC Berkeley, and Georgia Tech have published Evolving Infra, an open-source serving framework that adapts prefill-decode splits and KV cache eviction online without offline profiling, and benchmarked it against Nvidia Dynamo at production scale. Against that baseline, it reduced P95 time-to-first-token by up to 14× and mean end-to-end latency by up to 4.1×.
What the source says
The paper was submitted November 9, 2026 by Hao Kang, Weili Xu, and colleagues including Song Han (MIT Han Lab), Ion Stoica (UC Berkeley), and Tushar Krishna (Georgia Tech). Tests ran on up to 256 Nvidia H100 GPUs serving GLM-5.2 and Kimi K2.6 across coding, math, and computer-use workloads. The framework decouples compute from cache management, letting nodes switch between prefill and decode phases at runtime. An online planner sets the prefill-decode split; a distributed KV manager handles cache eviction and transfer using agent context signals. Code is open-sourced at github.com/mit-han-lab/EvolvingInfra.
Why it matters
Agentic inference produces irregular request patterns that break the assumptions baked into static serving configurations. Dynamo requires offline profiling to set those configurations; Evolving Infra adapts them at runtime, which is the core claim. A 14× P95 TTFT reduction at 256-GPU scale, on frontier models already in production use, is a meaningful signal that online adaptation is worth the engineering cost. The result is most relevant to teams running multi-step agent pipelines, where tail latency compounds across turns.