Abigail Jacobs

Come find us at COLM, I’ll be presenting this paper next Thursday afternoon (10/8, oral session 6 and poster session 6). Full paper here: arxiv.org/abs/2609.08812

What AI Benchmarks Actually Measure: Adapting Convergent and Discriminant Validity to Interrogate Fifty-Six AI Benchmarks arxiv.org

AI Weekly's analysis →

• A COLM 2026 paper runs 53 models through 56 capability and safety benchmarks and finds many benchmark labels do not match what the tests measure.
• BBQ-accuracy, assigned to the bias category, correlates more strongly with reasoning benchmarks than with other bias benchmarks in its own group.
• On safety concepts, correlations between model rankings on benchmarks sharing the same assigned concept are often weak, the authors report.

Read full analysis →

View on Bluesky →

To make our analysis possible, we collected model outputs and scores from 53 models on 56 capability and safety benchmarks. This data, including item-level model responses and score, is available here huggingface.co/datasets/mad...

madesai/what-ai-benchmarks-actually-measure · Datasets at Hugging Face huggingface.co

AI Weekly's analysis →

Read full analysis →

View on Bluesky →