Xu and Evans review maps four LLM roles in social science

A team led by Fengli Xu and James Evans has published a review in Nature Human Behaviour that sorts large language models into four distinct research roles for computational social science: 'subjects that simulate human behaviour, assistants that automate analysis, collaborators that generate hypotheses and evaluators that audit and improve research quality.' The analysis spans sociology, political science, psychology, anthropology and economics.

The paper documents LLMs simulating multi-agent societies with emergent phenomena, outperforming crowd workers on text annotation tasks, and conducting conversational interviews at scale. On X, James Evans framed the review as a thread on what the authors found, 'including where the field is fooling itself.'

The limits section is blunt. 'LLMs struggle with causal inference and raise urgent ethical concerns about privacy and misinformation, and their ability to reproduce social complexity remains uneven and unproven,' the authors write. They also flag sycophancy, unfaithful explanations in reasoning tasks, and an 'AI–AI bias' in which generated content drifts toward LLM-produced outputs.

Validation, the review concludes, is the central challenge for generative social simulation. That gap remains open even as agent populations scale into the millions.