Nature paper flags 504 fallacies in psychology measurement

Psychology scales that measure different things under the same name, and the same things under different names, are a long-running mess in the field. A paper in Nature Human Behaviour uses language-model embeddings to flag that mess at scale, identifying 504 candidate fallacies across 277 constructs in the International Personality Item Pool: 340 "jingle" cases where the same label hides different content, and 164 "jangle" cases where the same content wears different labels.

The authors, Dirk U. Wulff of the Max Planck Institute for Human Development and the University of Basel, and Rui Mata of the University of Basel, fine-tuned an MPNet model to place individual survey items, full scales, and construct labels into a shared vector space. Embedding similarity predicted internal consistency at r = 0.75 in-sample and r = 0.61 out-of-sample across 449 scales, with 81% of scales showing full structural recovery.

Running a clustering pass on the same embeddings, the authors show that collapsing the IPIP's 277 construct labels into 68 clusters produces a 58% reduction in fallacies. "Our approach tackles taxonomic incommensurability by boosting conceptual clarity and providing a more parsimonious taxonomy of constructs in psychology," the paper concludes. The fine-tuned model is posted on Hugging Face; two researchers in our tracker shared the paper's link.