Röttger et al. propose two subjective-NLP annotation paradigms

Paul Röttger, Bertie Vidgen, Dirk Hovy and Janet Pierrehumbert argue in a NAACL 2022 paper that datasets for subjective tasks like hate-speech classification conflate two incompatible goals, and that creators should pick a lane. They sketch a descriptive paradigm that 'encourages annotator subjectivity' alongside a prescriptive one that 'discourages it,' then run an annotation experiment on hate-speech data to illustrate the contrast.

The complaint is architectural. 'So far, dataset creators have acknowledged annotator subjectivity, but rarely actively managed it in the annotation process,' the authors write, adding that this 'has led to partly-subjective datasets that fail to serve a clear downstream use.' The two paradigms point at different downstream models: descriptive annotation 'allows for the surveying and modelling of different beliefs,' while prescriptive annotation 'enables the training of models that consistently apply one belief.'

The paper stops short of picking a winner. Instead the authors 'argue that dataset creators should explicitly aim for one or the other to facilitate the intended use of their dataset.' Two researchers we track have since surfaced it into our directory.