Excited to share our new paper, “What AI Benchmarks Actually Measure: Adapting Convergent and Discriminant Validity to Interrogate Fifty-Six AI Benchmarks,” accepted as an oral at COLM! arxiv.org/pdf/2609.08812
Who's Who of AI
Meera Desai
Why they matter
Researcher with public evidence across Models & releases, AI research, Evaluation & benchmarks.
- AI signals
- 3 past 30d
- Sources
- 2 distinct domains
- Discussões
- 0 past 30d
- Latest signal
- 22d ago
PhD student at University of Michigan School of Information. Data and evaluation practices for language models / language models as cultural technologies.
https://meera-desai.com
What they're sharing
arxiv.org
What AI Benchmarks Actually Measure: Adapting Convergent and Discriminant Validity to Interrogate Fifty-Six AI Benchmarks arxiv.org
madesai/what-ai-benchmarks-actually-measure · Datasets at Hugging Face huggingface.co
Articles & links
Come find us at COLM, I’ll be presenting this paper next Thursday afternoon (10/8, oral session 6 and poster session 6). Full paper here: arxiv.org/abs/2609.08812
AI Weekly's analysis
→
- A COLM 2026 paper runs 53 models through 56 capability and safety benchmarks and finds many benchmark labels do not match what the tests measure.
- BBQ-accuracy, assigned to the bias category, correlates more strongly with reasoning benchmarks than with other bias benchmarks in its own group.
- On safety concepts, correlations between model rankings on benchmarks sharing the same assigned concept are often weak, the authors report.
Read full analysis →
To make our analysis possible, we collected model outputs and scores from 53 models on 56 capability and safety benchmarks. This data, including item-level model responses and score, is available here huggingface.co/datasets/mad...
AI Weekly's analysis
→
Read full analysis →
Their network
In Meera Desai's orbit
Center = Meera Desai. Left = members they follow (green edges). Right = members who follow them (blue edges). Top = mutual follows (orange edges, slightly larger). Drag any node to reposition; click to open that profile.
Are you Meera Desai? Show it.
Add the Who’s Who of AI badge to your site or bio. It links back to this profile.
Markdown: [](https://aiweekly.co/whos-who/person/madesai-bsky-social)