TempCloze benchmark: Video-LLMs stumble on 'when', not 'what'

On a new video-cloze benchmark of 1,521 clips, proprietary Video-LLMs average 70.73% on identifying what event fills a missing middle segment but only 48.13% on placing it correctly in time, according to a preprint from Wenqi Pei and collaborators posted on arXiv and accepted to EMNLP 2026 Findings.

The benchmark, TempCloze, shows a model the beginning and end of a video and asks it to pick the true missing middle from a set of candidates. The distractors are pulled from the same source video and share scenes and objects, which the authors say is meant to block language and appearance shortcuts. Human accuracy sits at 97.00% across the three tested dimensions; random guessing is 25.00%.

The paper splits reasoning into three axes: "Semantic asks what event should happen, Alignment probes when it should occur, and Progression tests how it should unfold." Across 10 proprietary and 21 open-source systems, the pattern holds. "Many models identify Semantic content and event Progression," the authors write, "but temporal Alignment remains the bottleneck: models struggle to place the missing middle precisely between endpoints, while humans solve the task reliably."

The abstract reports aggregate scores by tier rather than naming which individual systems clear or miss the Alignment bar, and the authors extend the analysis with error-pattern and behavioural-sensitivity studies on two harder splits, TempCloze-Mixed and TempCloze-Hard.