GRUFF benchmarks LLM pronoun fidelity and bias in German

Large language models handle German masculine and feminine pronouns fine on their own. They slip on the neopronouns 'xier' and 'en,' and they lose the thread when another entity is mentioned in between.

That is the finding in GRUFF, a new large-scale German pronoun-fidelity dataset from Fabian Mewes, Anne Lauscher, and Vagrant Gautam, posted to arXiv on 28 May 2026. The task measures whether a model can 'correctly reuse a previously-specified pronoun for a discourse entity, independent of other potentially distracting discourse entities mentioned in between.' Prior work focused on English, which the authors describe as 'a language with limited grammatical gender and almost no gender agreement.'

GRUFF covers 'four different gender agreement systems in nouns, and four sets of pronouns.' The authors report that 'LLMs show strong grammatical agreement for masculine and feminine entities in the absence of explicit context, but not for neopronouns xier and en.' Encoder-only models are 'more robust in German than in English, reflecting the importance of grammatical gender.' Occupational stereotypes, they add, are 'poorly correlated across grammatical cases, and across most models, except ones with closely related architectures.'

The abstract names no specific model tested and no per-pronoun accuracy figures. The authors say they release 'all code and data to encourage further work on gender-inclusive language and referential reasoning in German.'