EvaOmni grades Omni LLMs on 1,027 evidence-grounded Q-A pairs

EvaOmni, assembled by researchers at Fudan University, Shanghai Jiao Tong University, Tsinghua University, Tencent Youtu Lab and several other Chinese institutions, scores Omni LLMs on 1,027 purely human-annotated question-answer pairs across 392 videos, and requires models to return not just an answer but the audio and visual time intervals supporting it. In the paper posted on Hugging Face, Gemini3.7-Flash topped the leaderboard with 79.4% accuracy, 70.8% Coverage Grounded Score, and 35.1% Bottleneck Grounded Score, the last metric penalizing weak temporal localization in either modality.

Humans sit far above the models: expert reviewers reach 94.3% B.G. on the same test. At Level 3 complex reasoning, Gemini3.7-Flash's accuracy stays close to its overall mark at 79.1%, but its B.G. collapses to 12.3%. The strongest open-weight entrant, MiMo-V2.5-310B-A15B, posts 56.8% accuracy and 20.5% B.G.; Qwen3-Omni-Instruct-30B-A3B scores just 0.4% B.G.

"Future models should not only pursue correct answers but also be able to precisely localize the evidence supporting their answers," the authors write, framing precise temporal grounding as "the capability current Omni LLMs most clearly lack." The release lands in a crowded stretch of multimodal evaluation work we've tracked over the past three months.