Embeddings flag 504 fallacies in psychology's scale taxonomy

Two researchers ran 4,452 items from the International Personality Item Pool through a fine-tuned language model and came back with 504 flagged fallacies: 340 cases where different measures share a single construct label, and 164 where similar measures live under different labels.

The paper appeared in Nature Human Behaviour. Dirk U. Wulff of the Max Planck Institute for Human Development and Rui Mata of the University of Basel built semantic embeddings for the 4,452 psychological items in IPIP, covering 459 scales, 277 constructs and 254 unique single- and multi-construct labels, then fine-tuned their model on 200,000 training examples drawn from paired personality items.

The authors describe the target plainly: "identifying jingle–jangle fallacies by identifying criteria that quantify high similarity in construct labels associated with low similarity in the associated scales—jingle fallacy—or, alternatively, low similarity in construct labels associated with high similarity in the associated scales—jangle fallacy."

Their embeddings predicted empirical internal consistency at Pearson r = 0.75 in sample and r = 0.61 out of sample. A structural recovery test returned 81% in sample and 79% out of sample across 443 scales. When the authors asked the system to propose a cleaner taxonomy, it produced a mapping with "58% fewer jingle–jangle fallacies while requiring 75% fewer unique labels." Two of the researchers we track have shared the link.

The paper does not claim to resolve the jingle–jangle problem. The authors conclude only that "semantic embeddings constitute a powerful tool for tackling taxonomic incommensurability in the psychological sciences."