ToolSpace Mines MCP Traces, Drops Frontier Models 38.2 Points

ToolSpace, a tool-use data-synthesis pipeline from researchers at Renmin University of China, Fudan University and Shanghai Jiao Tong University, raises the Qwen3.6-35B-A3B backbone from 23.0 to 41.0 average score across seven long-horizon agent benchmarks, reaching 43.2 after reinforcement learning. Its 744B variant posts 56.6 average, ahead of GPT 5.6 Sol at 48.9 and on par with Claude Opus 5, the paper reports.

The approach inverts the usual data-first pipeline for Model Context Protocol agents. Rather than inventing a database and then deriving tools from it, ToolSpace mines data, state and retry dependencies from adjacent calls in real MCP execution traces, then clusters the resulting weighted graph into environments. 'No LLM judgment is used,' the authors write; edges are typed by deterministic rules and weighted by how often one tool call actually followed another in a real trajectory. The reconstructed graph holds 77,760 tool-call-pattern nodes over 2,042 canonical tools, organized into 804 real MCP environments spanning Gmail, Google Sheets, Slack, Google Drive, Salesforce and Airtable. Data edges account for 69%, state for 25%, retry for 6%.

The same machinery was pointed at two existing benchmarks to produce AutomationBench Pro (300 tasks) and τ³-Bench Banking Pro (98 tasks), each reviewed by three doctoral-level experts over a two-day period with Fleiss κ=0.82 and 93.1% observed agreement. Seven frontier models drop by up to 38.2 points on AutomationBench Pro, with Claude Opus 5.5 falling 32.9 and GPT 6.1 Sol falling 26.1; on τ³-Bench Banking the drops land at 15.8 and 16.8 respectively. It arrives in a busy season for agent evaluation, with 526 prior agent stories logged on our agents stream over the past 90 days.

Average tool calls per task rise from 26.8-43.6 under Standard to 56.3-60.6 under Pro on AutomationBench, 'fully confirming that the added difficulty arises from longer and more frequent tool orchestration,' the paper states. The authors released the code, weights, and 34,376 SFT plus 653 RL samples from the pipeline.