Nature paper flags 504 fallacies in psych measurement scales

Two researchers ran a language model across 4,452 psychology test items and came back with 504 cases where either the same label covered different ideas or different labels covered the same idea.

The paper, published in Nature Human Behaviour by Dirk U. Wulff (Max Planck Institute for Human Development and University of Basel) and Rui Mata (University of Basel), goes after what psychometrics calls 'taxonomic incommensurability': the field's habit of giving different scales inconsistent names for what they measure. Wulff and Mata fine-tuned an MPNet embedding model on '200,000 pairs of personality items from several open datasets' and converted items, scales and construct labels into vectors they could compare directly.

What fell out when they pointed it at the International Personality Item Pool's 459 scales and 277 constructs: 340 'jingle' fallacies (same label, different content) and 164 'jangle' fallacies (different labels, similar content). The worst jingle label was 'organization' with 35 cases, followed by 'humor/playfulness' at 24 and 'toughness' at 23. On the jangle side, 'neuroticism' showed up 40 times, 'emotional stability' 23.

The authors then ran a clustering pass to clean up the taxonomy. It cut fallacies by 58% while collapsing unique labels by 75%.

On validation, the embeddings predicted internal-consistency scores at r = 0.75 in-sample and r = 0.61 out-of-sample, and reproduced 81% of scale structures fully. Wulff and Mata argue embeddings 'constitute a powerful tool for tackling taxonomic incommensurability' and can 'render these fallacies more transparent and increase conceptual clarity.'

The validation uses narrow datasets, the relabeling is sequential rather than holistic, and the method still needs expert agreement on how many constructs a domain ought to admit. A couple of researchers we track circulated the paper in the days after publication.