arxiv.org web signal

Claude Fable 5.2 tops TwinLab-Bench at 45.0 across 52 lab tasks

Anthropic Agents Safety ai-business

TL;DR

  • TwinLab-Bench pairs each of 52 laboratory tasks with its own 'TwinLab agent' that executes experiments grounded in a real lab's records.
  • Claude Fable 5.2 leads at 45.0 average, Anthropic's Claude Opus 5.5 trails at 40.2, and GPT-6.1-sol lands at 31.7 across 13 agents tested.
  • Authors Yang Wan and Linchao Zhu target the hardest of three abilities: an agent designing its own experiments, not just running or selecting them.

"We find that these tasks remain hard for frontier models," write authors Yang Wan and Linchao Zhu in a new arxiv preprint introducing TwinLab-Bench, a benchmark where the strongest system tested, Claude Fable 5.2, averages only 45.0 across 52 laboratory tasks spanning the physical sciences, life sciences and engineering.

What TwinLab-Bench tests is the hardest of three "increasingly open abilities" an AI scientist is expected to have: not just running a given computational workflow, and not just choosing the next measurement from a predefined design space, but designing its own experiments. Each task pairs the tested model with its own "TwinLab agent," which serves as the environment, reading the raw materials of the laboratory and calling calculators built from its records, then returning readings for whatever experiment the model designs. The TwinLab agent, the paper reports, "almost eliminates readings that deviate from the real records and inconsistent answers to a repeated request."

Thirteen agents were evaluated. Claude Fable 5.2 leads at 45.0, Anthropic's Claude Opus 5.5 trails at 40.2, and GPT-6.1-sol lands at 31.7; DeepSeek, Qwen, Kimi and GLM systems round out the table. The benchmark supports more than 100 experimental action types, and the agent's final report is scored against a hidden answer taken from real laboratory records.

The result arrives the same day Anthropic has been pushing the stronger narrative that its Claude models solved two open black hole problems. On TwinLab-Bench's harder test, the paper's next step, the authors write, is "an empirical study of where it fails."