Found first: a primary source the press has not covered yet.
Researchers from MIT, UC Berkeley, and Georgia Tech have released a paper describing Evolving Infra, an agent-adaptive LLM serving framework that reduces P95 time-to-first-token by up to 14× and mean end-to-end latency by up to 4.1× compared to Nvidia Dynamo. The framework adapts prefill-decode compute allocation and KV cache eviction at runtime without requiring offline profiling. It was submitted to arXiv on November 9, 2026, and code is open-sourced at github.com/mit-han-lab/EvolvingInfra.
What the source says
The evaluation runs on up to 256 GB300 GPUs serving two production-scale models, GLM-5.2 and Kimi K2.6, on coding, math, and computer-use tasks, with Nvidia Dynamo as the baseline. The core design decouples compute allocation from KV cache management. An online planner adjusts prefill-decode splits to match shifting workload patterns, while an agent-aware distributed KV manager handles cache eviction and transfer across nodes. Authors include Song Han (MIT Han Lab), Ion Stoica (UC Berkeley), Tushar Krishna (Georgia Tech), and Huizi Mao, among others.
Why it matters
Agentic workloads produce highly variable prefill-to-decode ratios and long, irregular context sequences that static serving configurations are not built for. Existing disaggregated systems, including Dynamo, allocate compute from offline profiles that go stale as workload patterns shift. Achieving up to 14× P95 TTFT improvement on production models at 256-GPU scale addresses a real bottleneck for teams running agentic inference at high concurrency. The open-source release gives infrastructure teams a concrete starting point rather than waiting for a vendor to absorb these ideas.