Xu-Evans Nature review: four LLM roles in social science

A Nature Human Behaviour review published on 26 October by Fengli Xu, James Evans and collaborators at Tsinghua, HKUST and Microsoft Research Asia sorts large language models in social science into four research roles: "subjects that simulate human behavior, assistants that automate analysis, collaborators that generate hypotheses, and evaluators that audit research quality."

The paper spans sociology, political science, psychology, anthropology and economics. The authors report that when prompted with demographic or ideological profiles, models "reproduce survey distributions, economic games and polarization dynamics," and that AI interviewers deliver protocols "with quality comparable to human interviews." Models that code qualitative data act, in the authors' phrasing, as "alien readers," surfacing patterns a human coder would miss.

The review is blunter about what the models can't do. "LLMs struggle with causal inference and raise urgent ethical concerns about privacy and misinformation, and their ability to reproduce social complexity remains uneven and unproven."

In a post on X marking publication, Evans said the thread covered "where the field is fooling itself." Two AI researchers we track shared the link.