Embeddings flag 504 jingle-jangle fallacies in psychology

Psychology has a label problem, and two researchers in Basel just quantified it. In Nature Human Behaviour, Dirk U. Wulff and Rui Mata report that running the items of 459 personality scales through a language-model embedding flagged 504 taxonomic fallacies: 340 scales sharing a label but measuring different things, and 164 measuring similar things under different labels.

The framing comes straight from the paper. "Psychological sciences are plagued by measures sharing similar labels yet capturing different constructs (jingle fallacy), or receiving different labels yet capturing identical constructs (jangle fallacy)," the authors write. Their fix is geometric rather than theoretical: represent each item, scale and construct label as a vector, then let distances in that space do the work that decades of nomenclature have not.

Applied to the International Personality Item Pool, the pipeline compressed 277 construct labels down to 68, which the authors report cut jingle-jangle fallacies by 58%. The embeddings also predicted empirical behavior of the scales, correlating with internal consistency at r = 0.75 in-sample and r = 0.61 out-of-sample. The authors conclude that "semantic embeddings constitute a powerful tool for tackling taxonomic incommensurability in the psychological sciences."

The paper was circulating among a couple of the researchers we track this week, which is unsurprising: the method is a clean example of using an off-the-shelf language model to audit a social-science literature from the outside. Whether the 68 consolidated labels survive expert pushback is a separate question the Nature piece does not try to settle.