EAR trims hate-speech bias in BERT without identity-term lists

A 2022 ACL Findings paper keeps coming up when teams ask how you strip identity-term bias out of a hate-speech classifier without maintaining a list of identity terms. The method, Entropy-based Attention Regularization (EAR), does the mitigation from inside the model rather than from a blocklist, and that framing is what makes it worth revisiting.

The problem the authors describe is familiar to anyone who has shipped a hate-speech classifier. Neural models get strongly influenced by identity terms like 'gay' or 'women' in the training data and start firing on them regardless of context, producing false positives and what the paper calls severe unintended bias. The usual fix is to hand the model a curated list of identity terms, or samples from the target domain, during training. That works, but it requires a-priori knowledge, and if an important term is neglected you have just introduced a new bias.

EAR takes a different route. The authors add an objective function that penalizes tokens with low self-attention entropy, which discourages the model from concentrating too hard on any single training-specific term. They fine-tune BERT via EAR on three benchmark corpora in English and Italian and report that the resulting model matches or exceeds state-of-the-art performance for hate-speech classification and bias metrics. Because there is no list to consult, the method also reveals overfitting terms — the tokens most likely to induce bias — which is a diagnostic byproduct a blocklist approach cannot give you.

The honest caveat is that the evidence base here is BERT plus three corpora in two languages. The abstract does not give you absolute deltas over baseline, guidance on how sensitive the entropy penalty is to hyperparameters, or any read on how EAR behaves inside larger decoder-only models where attention distributes very differently. Take the specifics as reported, not settled.

Still, the direction is useful. For trust and safety teams and for anyone auditing a classifier, an approach that reduces bias and also tells you which tokens the model was leaning on hardest is worth more than a mitigation that quietly hides the problem behind a list.