Embeddings flag 504 fallacies in personality scales

Psychologists have argued for a century that their field is drowning in duplicate and mislabeled constructs. A Nature Human Behaviour paper by Dirk U. Wulff of the Max Planck Institute for Human Development and Rui Mata of the University of Basel puts a number on the mess.

The authors fine-tuned MPNet on 200,000 personality item pairs and ran it over the International Personality Item Pool: 4,452 items, 459 scales, 277 construct labels. The embeddings flagged 504 candidate fallacies, split into 340 'jingle' cases (constructs sharing a label but measuring different things) and 164 'jangle' cases (different labels covering similar content). 129 of the 277 constructs were involved in at least one jingle fallacy; 80 in at least one jangle fallacy. Organization and orderliness alone produced 35 jingle instances.

The paper reports that 'embeddings can largely capture the psychometric structure of psychological scales and inventories,' predicting internal consistency at r = 0.75 in-sample and r = 0.61 out-of-sample, and recovering structural fidelity at 81% and 79%.

Clustering the embeddings collapsed the 277 labels into a 68-cluster taxonomy that cut the fallacy count by more than half while keeping semantic granularity, a 75% reduction in the number of unique construct labels required to describe the same inventories. The fine-tuned model is published as dwulff/mpnet-personality on Hugging Face.

The validation is narrow: five inventories (16PF, Big Five, FFM, HEXACO, NEO) inside one item pool, with no test of clinical or developmental measures and no expert-consensus layer on the proposed relabeling.