Evolving Infra paper reports 14× P95 edge over Nvidia Dynamo
TL;DR
- A new serving framework called Evolving Infra reports up to 4.1× lower mean end-to-end request latency and up to 14× lower P95 TTFT than Nvidia's Dynamo.
- Benchmarks ran on GLM-5.2 and Kimi K2.6 using up to 256 GB300 GPUs, per the arXiv preprint posted November 9, 2026.
- The core idea is decoupling compute allocation from KV cache management, with an online planner adjusting the prefill-decode split at runtime.
A new LLM serving framework called Evolving Infra cuts mean end-to-end request latency by up to 4.1× and P95 time-to-first-token by up to 14× over Nvidia's Dynamo, according to an arXiv preprint dated November 9, 2026. The benchmarks ran on GLM-5.2 and Kimi K2.6 using up to 256 GB300 GPUs.
The authors — Hao Kang, Weili Xu, Ion Stoica, Song Han and seven collaborators — pitch the design as a break from existing prefill-decode disaggregated systems that pin compute and cache splits at deployment. The framework instead rests on "decoupling compute allocation from KV cache management" and two moving parts at runtime: "an online planner matches the PD split to workload composition, while an agent-aware distributed KV manager guides cache eviction."
Dynamo is Nvidia's own production framework for LLM serving, which makes the comparison a head-to-head against the vendor's shipping baseline rather than a research prototype.
The abstract publishes no absolute latency figures, no per-workload breakdown, and no numbers against non-Nvidia stacks.
Originally reported by paper
Read the original article →Original headline: MIT-Led Open Framework Beats Nvidia Dynamo 14× on P95 LLM Latency for Agentic Workloads