Paun et al. compare six Bayesian annotation models in TACL

Paun et al. open with the problem crowdsourced NLP labels pose. Three things need identifying: the gold standard, annotator accuracy and bias, and item difficulty. The field had been reaching for different tools for each.

'Traditionally, majority voting was used for 1), and coefficients of agreement for 2) and 3),' the authors write in the TACL paper, published in 2018. Model-based analysis has since proven better on all three tasks, the paper reports, but comparisons on common datasets had been rare.

The authors run six Bayesian models against four datasets 'with varying degrees of noise in the form of random annotators,' varying how each model handles annotator ability, item difficulty, and parameter pooling (tying) across annotators and items. The abstract does not name the four datasets, and does not say which of the six models wins on which dimension.

The paper ends on something practitioners can use: 'guidelines for model selection, application, and implementation.' Two of the researchers we track posted the link in recent days.