GPT-4 matches expert forecasters on social experiment outcomes

A group at Stanford's AI for Public Benefit Lab reports in Nature that GPT-4, prompted to simulate a representative sample of American adults, predicted the outcomes of 469 experimental effects across 70 preregistered US survey experiments about as accurately as pooled human forecasters. The simulated treatment effects correlated with the actual measured ones at r = 0.85.

The part that pushes this past "the model is just retrieving what it read" is the subset of studies published after GPT-4's training-data cutoff. On those, the paper reports accuracy stayed high, with an r around 0.90. Something in the model's representation of how Americans react to framed survey questions is doing real predictive work, at least on this class of stimulus.

Why this matters if you don't run experiments for a living: expert forecasting panels and small pilot surveys are the two ways teams currently sanity-check what a randomized message test will do before spending on the real thing. If a language model can stand in for either of those with useful accuracy on the text-based survey questions that political campaigns, public-health comms, and marketing teams field every week, the cost of a "will this land?" pre-check drops sharply. The authors have put a live demo at treatmenteffect.app so you can try it against your own question.

The honest caveat, which the authors themselves flag, is that the model tended to overestimate effect sizes. So the ranking of which stimulus works best may be right while the magnitudes are inflated, which matters a lot if you use the numbers to size a rollout or a budget. The archive is 70 nationally representative US survey experiments with 119,330 participants, so nothing here tells you how the approach behaves outside the US, on non-text stimuli, or on behavioral outcomes rather than self-reports.

What the reporting doesn't give you is why the overestimation happens, or whether smaller and open-weight models replicate the result. But the direction is the part worth watching: research groups and applied comms teams now have a defensible reason to run cheap in-silico pilots before committing to human ones.