Gemini, mWhisper-Flamingo show face-based ASR bias up to 4.05 WER
TL;DR
- Swapping the face shown alongside identical audio shifts transcription accuracy by up to 4.05 word error rate points in mWhisper-Flamingo and Gemini.
- The gaps appear across self-declared gender, ethnicity, and their intersection, the authors report.
- The paper is described as the first bias evaluation of multimodal speech recognition.
Pairing the same audio clip with a different face changes how well multimodal speech recognition transcribes it, by up to 4.05 word error rate points, according to an EMNLP 2026 paper by Maya K. Nachesa, Vlad Niculae and Vagrant Gautam, which the authors describe as the first bias evaluation of its kind for this class of models.
The method is blunt. Take identical audio, use lip-sync to render it with faces drawn from different self-declared gender and ethnicity groups, and measure what the model writes down. The paper tests mWhisper-Flamingo and Gemini, and reports 'large quality-of-service differences across mWhisper-Flamingo and Gemini models, with drops of up to 4.05 word error rate points, across self-declared gender, ethnicity, and their intersection.'
The authors land on a flat warning. 'Providing more signals through additional modalities is not necessarily better, and may even lead to biased outcomes,' they write, calling on developers to 'evaluate, fix, and communicate such limitations.'
Shared on Bluesky by 2 AI experts
-
Vagrant Gautam @dippedrusk.com
A raciolinguistic finding that has lived rent-free in my head for years is: People rate the exact same English audio as less intelligible when shown a brown person's face rather than a white face. In work led by Maya Nac…
View on Bluesky →
Originally reported by aclanthology.org
Read the original article →Original headline: You Have a Face for Radio: Bias in Multimodal Speech Recognition Models