'Omni-Streaming Thinking' Paper Targets Cross-Modal Hallucinations

Streaming multimodal models have a specific failure: visual cues can fix an interpretation in memory before an utterance or sound event finishes, and later reasoning keeps relaying that fact even after audio contradicts it. The authors of a new arXiv paper on Hugging Face name this failure 'premature cross-modal commitment.'

"Visual cues often support an interpretation before an utterance or sound event is complete," they write. "If that interpretation enters memory as a fact, later reasoning can keep relaying it even after audio contradicts it."

Their proposal, Omni-Streaming Thinking (OST), marks each claim as pending, links it to a future verification interval, and stores audio and visual evidence separately. "When contradictory evidence is detected, a refutation process reduces the influence of the claim and its dependent states, and then guides a state update using the new evidence," the paper says. An answer gate decides when the pending claims are settled enough to speak.

The headline number: OST "outperforms the strongest open baselines on five streaming and audio-visual benchmarks by more than 10% relative on average," running a frozen Qwen3-Omni-30B-A3B-Instruct backbone with lightweight adaptation.

The authors also release OST-DiagBench, which holds video fixed and edits audio to probe agreement, absence, contradiction, coexistence, and subtitle-speech conflict. On that test, OST reaches d-prime 2.95; open baselines cap at 1.38. The comparison set is open-source only, and the diagnostic figure comes from a benchmark the same authors built — worth keeping in mind alongside the broader run of streaming and audio-visual work landing this quarter across our multimodal tracker.