Nature Human Behaviour web signal

Embeddings expose 504 jingle-jangle fallacies in IPIP scales

TL;DR

  • Embeddings of 4,452 IPIP items across 459 scales flagged 504 taxonomic fallacies: 340 share a label but measure different things, 164 vice versa.
  • Collapsing the pool's 277 construct labels to 68 model-derived clusters cut jingle-jangle fallacies by 58% while using 75% fewer labels.
  • A fine-tuned MPNet model predicted scales' internal consistency at r = 0.75 in-sample and r = 0.61 out-of-sample from the embeddings alone.

Running 4,452 items from the International Personality Item Pool through a fine-tuned language model, two researchers flagged 504 cases where personality psychology's labels do not match what the scales actually measure. Of those, 340 are scales that share a label but measure different things; 164 measure similar things under different labels. The work appears in Nature Human Behaviour.

The authors, Dirk U. Wulff of the Max Planck Institute for Human Development and Rui Mata of the University of Basel, state the problem plainly: 'psychology is plagued by the proliferation of measures that have been given similar labels yet may capture different constructs (jingle fallacy), whereas other measures have received different labels yet capture the same construct (jangle fallacy).'

The analysis covered 459 scales and 277 construct labels. Their example: 'organization' and 'orderliness' are near-synonyms by semantic similarity, but the scales carrying those labels pull on different item content, cleanliness in one and tradition in the other.

A clustering pass to clean up the taxonomy collapsed the 277 labels to 68 and cut jingle-jangle fallacies by 58%, with 75% fewer unique labels. Internal consistency of scales was predicted at r = 0.75 in-sample and r = 0.61 out-of-sample from the embeddings alone. The fine-tuned MPNet model and data are public, at Hugging Face and OSF.

The authors frame the method as general, calling embeddings 'a powerful tool for tackling taxonomic incommensurability in the psychological sciences,' with extensions suggested beyond personality to emotion, organizational, and learning research.

Shared on Bluesky by 3 AI experts