Transformer gains from demographic tuning vanish under controls
TL;DR
- The paper finds substantial gains when a multilingual Transformer is adapted to users' gender and age across English, German, French and Danish.
- Those gains do not stem from demographic knowledge once domain and language proficiency are controlled, the authors report.
- They conclude demographic specialization of pretrained language models remains an unsolved problem for modern NLP.
The headline on the paper flips mid-abstract. Writing in the ACL Anthology Findings of EACL 2023 proceedings, Chia-Chien Hung, Anne Lauscher, Dirk Hovy, Simone Paolo Ponzetto and Goran Glavaš test whether adapting a multilingual pretrained Transformer to a user's gender and age helps text classification. At first, yes: the authors report "substantial gains in task performance across four languages (English, German, French, and Danish)," matching older results from pre-Transformer NLP.
Then they control for domain and language proficiency, and the signal collapses. "Downstream performance gains from our demographic adaptation do not actually stem from demographic knowledge," they write.
The paper tries three common specialization methods, coupling continuous language modeling with multi-task prediction of demographic classes. The abstract names no per-task numbers and no datasets. Its verdict for the field is unusually flat: "demographic specialization of PLMs, while holding promise for positive societal impact, still represents an unsolved problem for (modern) NLP."
Shared on Bluesky by 2 AI experts
-
#MemoryMonday #NLProc Hung et al.'s 2023 paper, 'Can Demographic Factors Improve Text Classification?' finds demographic adaptations of Transformer NLP models don't notably boost performance. aclanthology.org/2023.findi…
View on Bluesky →
Originally reported by aclanthology.org
Read the original article →Original headline: Can Demographic Factors Improve Text Classification? Revisiting Demographic Adaptation in the Age of Transformers