Embeddings expose 504 jingle-jangle fallacies in psych scales

Two researchers fed 4,452 psychometric items from the International Personality Item Pool into a language model, and the resulting embeddings flagged 504 cases where psychology's measurement scales appear to be measuring the same thing under different names, or different things under the same name.

The paper, by Dirk U. Wulff of the Max Planck Institute for Human Development and Rui Mata of the University of Basel, was published in Nature Human Behaviour in March 2025. They fine-tuned an MPNet model on 200,000 personality item pairs, then mapped 459 scales covering 277 constructs into a shared vector space.

"Taxonomic incommensurability denotes the difficulty in comparing scientific theories due to different uses of concepts and operationalizations," the abstract states. The problem shows up in psychology as what the authors call jingle-jangle fallacies: of the 504 flagged cases, 340 were jingle (same label, different content) and 164 were jangle (different labels, same content).

The embeddings also predicted empirical scale reliability. Cronbach's α was recovered with mean absolute error of 0.026 to 0.029 in-sample and 0.065 to 0.070 out-of-sample. Internal consistency across items correlated at r = 0.75 in-sample and r = 0.61 out-of-sample. Scale structure was fully recovered in 81% of cases and partially in 95%.

The authors propose replacing the existing tangle with a taxonomy of 68 clusters, which they report reduces fallacy candidates by 58% while using 75% fewer construct labels. The fine-tuned model is published as `dwulff/mpnet-personality` on Hugging Face, and the authors claim the embeddings "can be used to predict empirical relations between measures, automatically detect taxonomic fallacies and suggest more parsimonious taxonomies."