'Self-Listening' Paper Targets Voice AI Anchor Interruption

Full-duplex voice models can listen and talk at the same time. That is the design point, and the source of a specific bug named in a new preprint from Xuanning Zhou and co-authors: "text generation, speech synthesis, and audio playback proceed asynchronously. As a result, what a model believes it has said may not match what has actually been played to the user."

The authors call the failure mode "anchor interruption": the model has to recover from a user cut-in while still tracking which of its own words the user actually heard.

Their fix, called Self-Listening, "interleaves user speech, model text, and the model's played speech." Feeding the realized audio output back into the model as an input stream, the authors write, "grounds interruption recovery in what the user has actually heard."

The team also publishes AnchorSpeech, a paired training and test set that "evaluates whether a model can respond consistently with the last completed item before an interruption." The abstract reports that models fitted with self-listening "achieve better anchoring performance" than full-duplex baselines. It names no baselines and reports no per-model numbers in the passage retrieved.