Claude Opus 5 tops VideoMemBench at 45.7%, authors report

The best-performing model on a new multi-turn video benchmark scores only 45.7%, according to the paper VideoMemBench, which tests whether multimodal language models can carry context across sustained conversations grounded in video. "Claude Opus 5 performs the best with an overall score of only 45.7%," the authors write, after evaluating more than 20 open- and closed-source MLLMs.

The dataset covers 326 videos and 2,476 dialogue turns, of which 1,387 turns carry cross-turn dependencies. The paper separates "episodic dependencies," where a later turn references visual content seen earlier, from "answer dependencies," where a later turn builds on a prior model response. Dependencies were validated for necessity and non-leakage: a flagged question has to be unanswerable without the prior turn, and the prior turn cannot restate the answer.

Construction used a three-role LLM pipeline (planner, generator, judger) followed by human expert review. The abstract does not publish per-model scores for other frontier systems or break performance down by dependency type.