I'll be at #EMNLP2026 --- looking forward to (re)-connecting with folks! Ben will be presenting our work on the automated reproduction of social science results (poster Tuesday at 9:00) arxiv.org/abs/2604.21965
- A new study tests whether LLM agents can reproduce 48 empirical social-science papers using only a methods section and the original data, with no access to code.
- Agents ran under strict information isolation and were graded by cell-level comparison against the published results.
- Agents largely recovered published results, but performance varied, and failures came from both agent errors and papers that underspecified their own methods.