Video-Oasis audit: 55% of video benchmarks need no video

A quiet finding from an ECCV 2026 paper probably says more about the last two years of video-LLM leaderboards than any of the winners themselves. Geuntaek Lim and colleagues report that 55% of samples in existing video understanding benchmarks are solvable without any visual input or temporal context. Not most of the hard cases, more than half of everything.

The follow-through is the interesting part. When the authors strip out those shortcut-solvable samples and re-score, state-of-the-art models perform only marginally above random guessing on what is left. Which is to say, the visible margin between the top video model and a coin flip on the genuinely video-native questions is small enough that the public rankings we have been citing may be measuring language priors and world knowledge rather than anything that reads pixels.

Why this matters if you are not training video models yourself: the leaderboards those models climb are the ones enterprise buyers, product teams, and analysts have been using to pick a video Q&A, captioning, or moderation vendor. If the ranking is dominated by which model best guesses from a text question alone, procurement decisions have been anchored on the wrong signal. The authors' framing is deliberately not another benchmark; they position Video-Oasis as a diagnostic suite for systematically auditing existing video understanding benchmarks, which is a more useful contribution to the field than launching yet another leaderboard.

The honest caveat is what a single-paper claim can and cannot settle. The write-up quantifies a shortcut problem and a post-filter collapse, but it does not tell you which specific benchmarks contribute most to that 55%, whether video-native training regimes resist the collapse better than image-first fine-tuning, or what a cleaned leaderboard looks like ordered from top to bottom. Those are the questions I would want answered before rewriting a model shortlist.

The direction is the part worth watching. If audit frameworks like this get adopted, the labs that win are not the ones with the flashiest headline score, they are the ones whose gains survive the filter. That could reward more video-native training strategies over scale-only VLM extensions, and it gives buyers a cheap sanity check they did not have before.