Max Planck pair use embeddings to flag 504 psych scale fallacies

In a new Nature Human Behaviour paper, Dirk Wulff and Rui Mata use semantic embeddings from a fine-tuned MPNet model to catalogue overlap across 4,452 items and 459 psychological scales in the International Personality Item Pool, flagging 504 candidate jingle-jangle fallacies: 340 jingle and 164 jangle.

The pair, based at the Max Planck Institute for Human Development and the University of Basel, state the motivation plainly. "psychology is plagued by the proliferation of measures that have been given similar labels yet may capture different constructs (jingle fallacy), whereas other measures have received different labels yet capture the same construct (jangle fallacy)," they write.

The embeddings predicted empirical internal consistency (Cronbach's α) at a Pearson correlation of r = 0.75 in sample and r = 0.61 out of sample for the 449 scales with empirical scores available. Structural properties were fully recovered for 81% of scales in sample and 79% out of sample, with 95% partial recovery in both.

The authors then propose a 68-cluster alternative taxonomy against the original 150, reporting "58% fewer jingle–jangle fallacies while requiring 75% fewer unique labels." The model behind the analysis was fine-tuned on 200,000 personality item pairs.