Embeddings flag 504 jingle-jangle fallacies in IPIP scales

Two psychologists used a language model to put 4,452 psychometric items into a shared vector space and came back with a list of 504 candidate 'jingle-jangle fallacies' across the International Personality Item Pool: 340 jingle, 164 jangle.

The paper, by Dirk U. Wulff and Rui Mata in Nature Human Behaviour, opens on the methodological problem head-on. "Taxonomic incommensurability denotes the difficulty in comparing scientific theories due to different uses of concepts and operationalizations," the authors write. Their fix is to represent items, scales and construct labels as embeddings in the same vector space, then let cosine similarity arbitrate what is actually the same thing.

The scale is concrete. The embeddings cover 4,452 items across 459 scales and 277 constructs. Predicted internal consistency tracks observed consistency at r = 0.75 in-sample and r = 0.61 out-of-sample. For structural fidelity, 81% of scales are recovered in-sample and 79% out-of-sample, with 95% at least partially recovered.

The parsimony result is the one worth sitting with. Collapsing the IPIP's 277 construct labels into 68 model-derived clusters produces 58% fewer jingle-jangle fallacies while using 75% fewer labels. The implication is uncomfortable for anyone building psychometric assessments on top of the existing taxonomy: a large slice of the vocabulary may be labelling the same underlying thing in different clothes.