TAO cuts agentic LLM TTFT up to 36.3% with tool-aware scheduling

Across 12 model-workload-concurrency settings, a new scheduling framework called TAO reports cutting mean time to first token by up to 36.3%, lifting decode throughput by up to 29.5%, and reducing latency by up to 26.8% against the best state-of-the-art baseline on each metric. The numbers come from an arXiv paper submitted October 27, 2026 by Shuhuai Li, Shengdong Ye, Dongdong Ge, Yinyu Ye and Jianghao Lin.

The authors describe the gap plainly. "Agentic services alternate between model inference on clustered GPUs and external tools on distributed CPUs," they write, and existing systems "overlook tool-return readiness in service evaluation, wasting limited resources," while leaning on "heuristic strategies to decide admission and placement on GPUs, resulting in suboptimal scheduling decisions."

TAO's answer is a value interface that mixes "near-term tool readiness, node-specific cache credit with time-aware decay, and token footprint," fed to a dynamic program that jointly selects and places waiting sessions under GPU capacity constraints.

Every headline figure is an "up to" drawn from those 12 reported settings, not an average. The abstract names no baselines, no model sizes and no hardware.