Gemini-3-Pro scores 66.4/100 on live-video assistant benchmark

Gemini-3-Pro reaches 66.4 out of 100 on OmniAssistBench, a new interactive-video benchmark that tries to score omni-modal models as live assistants rather than passive clip watchers. The open-source Qwen3-Omni-Instruct comes in at 51.2. The paper on arXiv, led by Xianyun Sun and Chaoyou Fu, argues that static datasets cannot capture assistant work because "the model's unpredictable response dynamically changes the user's subsequent actions, which static offline datasets cannot accommodate."

To get around that, the authors reverse-engineered existing Internet videos into multi-turn clips with deduced user goals, then fed models predefined priors so different runs could be compared along the same route. The pipeline, they write, "required over 1000 expert person-hours to build the dataset."

Three failure modes recur. Models struggle with visual prompts such as hand gestures, fail to maintain historical context during multi-turn interactions, and fail to delay a response until the target event occurs. The abstract's own read is blunt: current systems "frequently provide incorrect or incomplete answers," and the results "indicate substantial room for improvement before models can become reliable assistants."