Wallach team: GenAI evaluation is a measurement challenge

Current measurement tasks for evaluating generative AI systems 'lack sufficient scientific rigor.' That is the opening claim of a position paper from twenty researchers, led by Hanna Wallach, in the ICML 2025 proceedings.

The authors cite what one commentator described as 'a tangle of sloppy tests [and] apples-to-oranges comparisons.' Their argument is that the ML community 'would benefit from learning from and drawing on the social sciences when developing and using measurement instruments for evaluating GenAI systems.'

The contribution is a four-level framework 'grounded in measurement theory from the social sciences,' aimed at measuring 'concepts related to the capabilities, behaviors, and impacts of GenAI systems.' Two implications follow. One is that evaluation becomes less of an ML-only room: stakeholders from outside the field can engage on conceptual grounds, debating what a given test is actually trying to measure. The other is more rigor at the operational layer, through what the authors call 'a set of lenses for interrogating validity.'

The abstract itself does not enumerate the four levels. Two of the AI researchers in our Who's Who directory shared the paper's link.