Embeddings flag 504 'jingle-jangle' errors in psych measures

A pair of psychologists trained a language model on 200,000 personality-item pairs and pointed it at the International Personality Item Pool, a public archive of 4,452 items organised into 459 scales covering 277 distinct constructs. The model flagged 504 candidate 'jingle-jangle' fallacies: 340 cases of identical labels hiding different measures, 164 cases of different labels wrapping the same measure. 129 of the 277 constructs turned up in at least one jingle fallacy.

Dirk Wulff of the Max Planck Institute for Human Development and Rui Mata of the University of Basel, writing in Nature Human Behaviour, describe their approach as 'a powerful tool for tackling taxonomic incommensurability in the psychological sciences.' Their parsimonious clustering compressed 277 constructs into 68 labels, which the paper reports as yielding '58% fewer jingle-jangle fallacies' and requiring '75% fewer unique labels.'

The model is an MPNet fine-tuned on personality-item pairs and posted to Hugging Face as dwulff/mpnet-personality. It predicted scale internal consistency at r=0.75 in-sample and r=0.61 out-of-sample, and fully recovered 81% of scale structures in-sample. Two researchers we track in the AI Who's Who shared the paper's link around publication.

The paper reports no human-rater check on whether psychologists agree with the proposed merges.