METR update: AI coding doubles fast, judgment work lags
TL;DR
- METR's Time Horizon 1.1 expands the benchmark from 170 to 228 tasks, adding judgment-heavy work like designing AI evaluations.
- Software-engineering time horizons have doubled roughly every seven months since 2021; judgment tasks improve more slowly.
- No model produced an AI evaluation METR employees deemed useful; Opus 5.5 was only 'borderline above the bar.'
METR's updated time-horizon benchmark, released as Time Horizon 1.1, expands the suite from 170 to 228 tasks and for the first time scores models on judgment-heavy work — including a task that asks models to design new AI evaluations METR could actually use. MIT Technology Review reports that on the well-defined coding and math side, the exponential still holds: time horizons have been doubling roughly every seven months since 2021. On the judgment side, the picture is less flattering.
No model produced an AI evaluation that METR employees considered useful. Opus 5.5, one of the newest systems tested, reached what METR calls "borderline above the bar" on the eval-design task, which the organization describes as not yet usable for its research but promising. On a second judgment task — scoring philosophical arguments — the models landed below human performance, though METR staff themselves disagree about how relevant that test is to automating AI research.
"By their nature, hard-to-verify or judgment-laden tasks are much harder to study, and especially to quantify," METR's Matt Putz told the magazine. Computer scientist Sayash Kapoor framed the stakes more sharply, arguing that "research taste" — choosing promising questions, hypotheses and experiments — "could be a major bottleneck" on the path to automated AI research.
The update lands into a week of coverage where frontier agents keep showing the same shape: strong on bounded tasks, brittle on open-ended ones. METR itself cautions that its own results above 16 hours remain unreliable, a reminder that the plot's most eye-catching numbers sit inside the error bars rather than above them.
Originally reported by technologyreview.com
Read the original article →Original headline: METR Expands Time Horizon Plot, Finds Frontier Models Flunk at Designing AI Evaluations