Wulff and Mata use embeddings to flag 504 psychology fallacies

Two psychology researchers fed the International Personality Item Pool into a fine-tuned language model and came back with 504 cases where the field is either calling the same thing by different names or different things by the same name.

The paper by Dirk U. Wulff and Rui Mata, published in Nature Human Behaviour, names the problem up front. "Taxonomic incommensurability denotes the difficulty in comparing scientific theories due to different uses of concepts and operationalizations," the authors write. Their tool was a fine-tuned MPNet embedding model, "fine-tuned using 200,000 pairs of personality items from several open datasets," pointed at 4,452 items drawn from 459 scales and 277 constructs in the IPIP.

The authors report that they "identified a total of 504 fallacy candidates, specifically, 340 jingle and 164 jangle fallacies." Predicting internal consistency from the embeddings alone produced "a strong Pearson correlation between the observed and predicted internal consistencies of r = 0.75 (in sample) and r = 0.61 (out of sample)." When they let the embeddings regroup the inventory, the result was "58% fewer jingle-jangle fallacies while requiring 75% fewer unique labels." That comes to 68 construct labels replacing the 254 the IPIP currently implies.

Two researchers on our tracker flagged the paper. The fine-tuned model is public at dwulff/mpnet-personality.

The paper's limitations are stated plainly. Validation rests on a bounded dataset, relabeling is sequential rather than holistic, and moving from 254 labels down to 68 still requires expert consensus on the right number of constructs. The 68-label scheme is the authors' proposal, not psychology's verdict.