Pew Finds AI Survey Stand-Ins Miss Humans by 12 Points

Pew Research Center ran its most direct test yet of whether AI can substitute for human survey respondents and the answer is no: across nearly 300 questions, AI-generated digital twins of real panelists diverged from their human counterparts by an average of 12 percentage points.

The Data Labs post, published September 30 and authored by Athena Chapekis, Arnold Lau, Samuel Bestvater, Sono Shah, Andrew Mercer, and Aaron Smith, used Anthropic's Claude Opus 4.6 to answer three American Trends Panel waves from early 2026, Waves 185, 190 and 192. Each AI respondent received demographic details and political-typology responses for a specific real panelist. Roughly 28% of questions produced errors above 15 points. Nearly half the questions had at least one answer choice that was not selected by a single AI-generated respondent.

The subgroup picture was worse. Average error ran 16.1% for Republicans and Republican-leaners, 15.1% for Black adults and 13.9% for Hispanic adults. On individual items the model put Trump job approval at 46% versus 34% among humans, said 97% of Hispanic respondents would likely follow the World Cup versus 43% actual, and reported awareness of data centers at 3% against the human reading of 25%. On a First Amendment knowledge question, 98% of AI respondents answered correctly versus 52% of real Americans. The model also ducked uncertainty: human panelists were "around four times as likely as the AI model" to pick a 'not sure' option where offered.

Pew did not soften the implication. "AI polling is not a replacement for rigorously surveying real humans," the authors write, and the Center "has no current or future plans to use AI models to generate survey results." They close with the line that "there is no substitute for rigorous, multimode, probability-based surveys that allow real members of the public to speak their minds on issues of importance."