Hume Paper: 6 of 11 ASR Models Reproduce Benchmark Text Verbatim

Six of eleven widely used open-source ASR models reproduce verbatim reference transcripts from public benchmarks even when the audio contradicts, masks, or ambiguously supports them, according to a new paper from Hume AI researchers posted to arXiv on August 20. The tested lineup, laid out in a companion Hugging Face blog post, spans speech systems from Nvidia, Microsoft, IBM, Cohere Labs, Bosonai and Qwen alongside OpenAI Whisper, Mistral's Voxtral-Mini and Moonshot's Kimi-Audio.

The Hume team, led by Theo Lebryk, probed models with three behavioral tests: reference disagreement, masked-number recovery, and orthographic switching. On a VoxPopuli recording where a speaker audibly says a courtesy "Thank you" that the benchmark reference had omitted, six of the eleven models likewise dropped the phrase. Roughly 40% of VoxPopuli test clips carry reference errors, about 3% of all reference words, and the affected models reproduce those errors between 18% and 30% of the time.

The other probes push harder. Top systems recover masked numbers on LibriSpeech at rates of 30 to 40%, and some clear 90% accuracy when asked to choose between two written forms that sound identical, matching the spelling convention of the benchmark they think they are being tested on. "Their scores may improve because they have learned benchmark-specific patterns and not because they have become better at the underlying task," the authors write. In the abstract, they add that models "respond to narrow acoustic cues to override the faithful representation of the audio in favor of a benchmark-optimized policy," a behavior they show can be triggered or suppressed through low-rank linear steering.

Hugging Face has now added a "Benchmark fitting" tab to its Open ASR Leaderboard that tracks VoxPopuli reference-error rates and orthographic switching across public datasets. The paper stops short of publishing a per-vendor composite ranking of which of the eleven models is worst hit.