WhatWorkedBench Measures AI Agents' Experimental Understanding

AI research agents can inspect code and launch experiments; a new benchmark asks the flatter question of whether, after a limited number of measurements, they can predict which component configurations will actually win. WhatWorkedBench, posted to arXiv on 23 September by Jingjie Ning, Xueqi Li, Yibo Kong and Dongting Li, sets that up formally.

The task is spelled out in the abstract: agents "inspect code, select measurements, and submit a response surface, a table predicting scores for every configuration of component settings." Exhaustive CPU runs supply reference effects for changing each component while holding the others fixed. The scale is not tiny — 36 tasks from 30 data sources and 8 workflow types, 1,248 configuration records, 4,206 numerical-control records, and 108 agent episodes across the original six workflows.

The results are uneven, and the paper is candid about it. "At eight new measurements, pair-effect ridge selects an optimum on 15 of 22 sources and limits every effect error to 10% of score range on three." Fitting a Gaussian process on the same agent observations lifts effect recovery from 0.632 to 0.698 in what the authors call "the original Flash cohort," and from 0.621 to 0.720 in an additional cohort. On six completed beat-detection and graph submissions, the same-observation GP raises family-macro recovery from 0.303 to 0.455. On six workflows with six binary options at 20 new measurements, encoding code equivalences — configurations with identical behavior — pushes GP recovery from 0.248 to 0.462.

The abstract does not name the models behind the "Flash cohort," list institutional affiliations, or report per-task numbers. It arrives during a busy week for agent research on our feed, alongside work like EmbodiedSWE on turning coding agents into training-data generators.