Wulff-Mata paper flags 504 jingle-jangle errors in psych scales

Psychology has a vocabulary problem. Two researchers ran the International Personality Item Pool, 4,452 items spread across 459 scales, through a fine-tuned language model and flagged 504 places where the field's measurement labels appear to describe the same thing twice or different things once, according to a paper by Dirk Wulff and Rui Mata in Nature Human Behaviour.

The underlying issue is old. 'Taxonomic incommensurability denotes the difficulty in comparing scientific theories due to different uses of concepts and operationalizations,' the authors write. Two scales with different names can measure the same construct; two with the same name can measure different ones. Wulff and Mata turn that into a geometry problem: embed every item, scale and construct label in a shared vector space and read off the distances.

Of 277 construct labels examined, 129 were involved in at least one 'jingle' fallacy (same label, different thing), and 80 in a 'jangle' fallacy (different labels, same thing). A clustering pass collapsed the 277 labels into 68, cutting the fallacy count by 58% and requiring 75% fewer unique names. The embedding model also predicted scale correlations with mean absolute errors between 0.037 and 0.047, and fully recovered 81% of the original scales.

'These findings suggest that semantic embeddings constitute a powerful tool for tackling taxonomic incommensurability in the psychological sciences,' the authors conclude. Two of the researchers on our Who's Who watchlist had posted the link by the time we logged the alert.