TAO Scheduler Pauses LLM Agents at Tool Calls, Cuts Latency 26.8%

Pause an LLM agent session the moment it calls an external tool, let the GPU serve someone else while the tool runs on CPU, then re-admit the paused session when the tool is close to returning. That is the pitch of TAO, a scheduler five AIOR-Research authors (Shuhuai Li, Shengdong Ye, Dongdong Ge, Yinyu Ye, Jianghao Lin) posted to arXiv on October 27.

The abstract lays out the gap plainly. "Agentic services alternate between model inference on clustered GPUs and external tools on distributed CPUs. In this scenario, existing systems (1) overlook tool-return readiness in service evaluation, wasting limited resources, and (2) use heuristic strategies to decide admission and placement on GPUs, resulting in suboptimal scheduling decisions." TAO's "value interface combines near-term tool readiness, node-specific cache credit with time-aware decay, and token footprint," and the paper's dynamic program "jointly selects and places waiting sessions under capacity constraints."

On DeepSeek-V4 Flash, Qwen3-32B and GLM-4.7, across 12 reported model-workload-concurrency settings, the authors claim up to 26.8% lower latency, up to 36.3% faster mean time-to-first-token and up to 29.5% higher decode throughput, with scheduler overhead under 0.12%. The tool-return readiness predictor is reported at 0.410 s mean absolute error on 10,967 tool calls.

The "up to" is doing work. Twelve setting-level numbers across three models are best cuts across configurations, not a flat guarantee any single workload will see. The paper lands amid a run of agent-serving coverage on our tracker, including 120 inference stories in the last 90 days.