Auto-psych: AI agents run end-to-end psychology research

A new preprint called auto-psych hands the messy parts of cognitive psychology to AI agents, from coming up with theories to running experiments on real people online. The paper, by Ben Prystawski, Kushin Mukherjee, Daniel Wurgaft, Linas Nasvytis, Michael Y. Li, Noah D. Goodman, and Michael C. Frank, posted to arxiv, describes a system that uses agents to "generate hypotheses, design experiments, and analyze data" inside a closed discovery loop.

The architecture is two nested loops. The inner one generates probabilistic cognitive models, fits them to data, and looks for weaknesses. The outer one designs and launches online experiments, recruits human participants, and feeds the results back. The test bed is a classic problem in cognitive psychology, the way people perceive randomness in coin flip sequences, which has the useful property that there is existing theory to beat.

The headline claim is that in three independent human experiments, the theories auto-psych came up with showed superior fit compared to "theories generated from the scientific literature." The authors also report recovering ground-truth theories from synthetic data, and they say the nested structure was essential, removing it tanks performance.

The honest caveat is that this is one narrow task domain, and the comparison is against whatever theories the authors selected from the literature, not against a fresh human researcher attacking the same problem. What the reporting doesn't give you is which underlying model the agents were running on, how many participants each loop recruited, or what discovery cost in compute and crowdsourcing per theory. Those numbers will matter a lot for whether other labs can actually use this.

If the loop generalizes beyond toy domains, the interesting consequence is who gets to do this kind of science. Computational cognitive psychology has long been bottlenecked by researcher time on the theory-building step. An agent loop that proposes, tests, and revises against real participants moves that bottleneck somewhere else, probably to evaluation and replication.