LLM agents reproduce 48 social-science papers from methods alone

LLM agents can reproduce the results of empirical social-science papers given only the methods section and the original data, according to a new arXiv preprint from Benjamin Kohler, David Zollikofer, Johanna Einsiedler, Alexander Hoyle and Elliott Ash.

The system extracts a structured methods description, hands it and the original data to an agent, and compares the output cell by cell against the published numbers. The agents never see the original code, results, or paper.

"Recent work uses LLM agents to reproduce empirical social science results with access to both the data and code," the authors write. "We broaden this scope by asking: Can they reproduce results given only a paper's methods description and original data?"

The study runs 48 papers with human-verified reproducibility through four agent scaffolds and four LLMs. The authors report that agents "can largely recover published results," but add that performance "varies substantially between models, scaffolds, and papers." The root cause analysis points two ways: "failures stem both from agent errors and from underspecification in the papers themselves."

The abstract does not publish per-scaffold or per-model accuracy, and the public page does not name which four LLMs were tested. The paper appears at EMNLP 2026 in Budapest, pages 9939–9980. Two researchers in our tracked directory had shared the arXiv link by the time we picked it up.