Wulff and Mata use embeddings to flag 504 psychometric fallacies

Two researchers took every item from one of personality psychology's most-used instrument pools, ran them through a fine-tuned language model, and asked the embeddings which labels were doing honest work. The answer, published in Nature Human Behaviour by Dirk U. Wulff and Rui Mata: not many.

Across 4,452 items spanning 459 scales and 277 construct labels in the International Personality Item Pool, the method surfaced trouble by the hundred. "Applying this heuristic criterion, we identified a total of 504 fallacy candidates, specifically, 340 jingle and 164 jangle fallacies," the paper reports.

The authors describe the problem in plainer terms earlier on. "These terms suggest that psychology is plagued by the proliferation of measures that have been given similar labels yet may capture different constructs (jingle fallacy), whereas other measures have received different labels yet capture the same construct (jangle fallacy)."

Then they rebuild. A parsimonious relabeling using 68 clusters produces "228 fallacies (215 jingle and 13 jangle fallacies)," and the authors summarize the exercise as "resulting in a streamlined mapping incurring 58% fewer jingle–jangle fallacies while requiring 75% fewer unique labels."

The predictive side holds up away from training data. Observed and predicted internal consistencies correlate at r = 0.61 out of sample. Seventy-nine percent of 443 scales are fully recovered out of sample. Scale-correlation mean absolute errors sit between 0.037 (BIG5) and 0.047 (16PF) in sample.

The validation set is narrow: the 16PF, HEXACO, NEO and BIG5 instruments. The authors are careful that their results may not generalize to the broader psychological literature. A pair of researchers we track circulated the paper after publication, and it fits a growing pattern of language models being turned on a discipline's own concept hygiene.