Lambert dissects Artificial Analysis v5 and its trade-offs

Artificial Analysis released v5 of its Intelligence Index with four categories weighted evenly at 25%: Knowledge Work, Coding, Science, and General. Inside each slice, two or three benchmarks carry the water: AA-Briefcase at 15%, AutomationBench-AA at 5% and GDP.pdf at 5% for Knowledge Work; FrontierCode 2.0 at 15% and Terminal-Bench 4.0 at 10% for Coding; Terminal-Bench-Science at 15% and a CritPt v2 hard subset at 10% for Science; AA-Omniscience v1.1 at 15% and HLE-Diamond at 10% for General.

Nathan Lambert, writing on Interconnects, reads the update as the latest round of what he calls the benchmarker's dilemma. "Everyone building a benchmark faces a sort of paradox," he writes, and "every method of aggregating scores comes with meaningful trade-offs." The pressure to collapse diverse capabilities into one rankable number is what makes an index useful and what makes it misleading at the same time.

The stakes he puts on it are blunt. "You dictate the winners and losers of a trillion dollar market," Lambert writes of the benchmark operators. "You also influence some of the models themselves." Smaller labs, he argues, remain existentially dependent on strong index rankings for credibility, while Anthropic and OpenAI shifted focus from public benchmark performance months or years ago, prioritizing internal use cases instead.

Lambert names Epoch AI's Capabilities Index as a distant second, with CursorBench, Vals AI, and Intelligence.ai as growing alternatives. On the open side he reads coding as the lone frontier where open models stay competitive, and expects "the next major model cycle, say in 2-3 months with GLM 6 or Kimi 3.5" to tighten the gap. "Hill-climbing is easy," he writes. "Knowing what to hill-climb is hard."