Oh and Gobet: 1B Models Match 70B on Trained Cognitive Tasks

A new arXiv paper makes a surprising claim about how much scale a language model actually needs to stand in for a human research subject. Nick Oh and Fernand Gobet trained fourteen models ranging from 135M to 14B parameters on Psych-101, a dataset of 10.7 million trial-level choices drawn from 160 experiments, and report that models in the 0.6B to 1B range performed comparably to a 70B baseline on held-out participants. For in-distribution simulations, they write, 'scale barely matters.'

The catch sits right next to that headline. Once the task structure moves outside the training distribution, the scaling picture flips: the authors describe 'a markedly steeper scaling gradient' outside the training set, meaning larger models start to pull ahead where the small ones cannot follow. So a sub-1B model is a plausible cognitive proxy for replicating known experimental paradigms, but not obviously for piloting genuinely new designs, which is where a proxy would be most useful.

A separate result attempts to close off a common critique of behavioral AI work, that models secretly win by memorizing choice histories rather than actually processing stimuli. Oh and Gobet report that masking the content of stimuli and feedback destroys 75.7% of the learned information, which they take as evidence the models are leaning on what they see rather than just the sequence of past responses. They also find models show invariance on tasks with independent trials but sensitivity where trial order depends on prior responses.

The abstract does not name the specific 70B baseline used for comparison, does not break down which task families collapse hardest out-of-distribution, and does not translate the 75.7% information-loss figure into a downstream accuracy hit. Those gaps matter for anyone weighing whether to substitute a small model for participant recruitment in a real study.

Still, for a lab that wants to sanity-check a well-known paradigm on a laptop rather than pay for a 70B inference budget, this is a useful data point. Cheap cognitive proxies for the paradigms we already understand, and human participants where the design is new, looks like a workable division of labor.