GST-Bench: top VLM scores 42.68 vs human 79.08 on video maps

A new video benchmark on arXiv is worth flagging because it isolates something the current wave of vision-language models is quietly bad at, and the gap is not small. On GST-Bench, 22 frontier VLMs were asked to build a globally consistent picture of a scene from continuous video. The strongest zero-shot model landed at 42.68. Humans doing the same task scored 79.08.

The setup is the interesting bit. The authors built the benchmark from 6,790 minutes of synthetically generated video with human-verified questions, and the tasks force the model to reason from novel viewpoints and translate egocentric observations into something like a top-down representation. That is a different job from the local, single-frame spatial reasoning most existing evals measure, and the paper's framing is that today's models can do the local piece and then fail to stitch those observations together over the full clip.

Why this matters if you are building on top of these models: embodied agents and robotics systems increasingly assume the VLM can carry a stable mental map of an environment as the camera moves. If the best frontier system tops out at roughly half the human score on that specific competence, product roadmaps that lean on it for navigation, memory of what was seen earlier in a clip, or reasoning about a space the camera has not directly re-observed all inherit that ceiling. The authors also release GST-Train, a companion dataset positioned as a resource for future research on the same problem, which signals the intended fix is more targeted training data rather than a purely architectural change.

The honest caveat is that this is a single preprint on a synthetically generated corpus. The authors report a 36-point human-model gap across 22 systems, but what I retrieved does not break down which specific models were tested, how the gap scales with clip length, or whether fine-tuning on GST-Train actually closes the gap rather than narrowing it. Take the numbers as reported, not as settled.

The forward-looking read is that this is exactly the kind of eval that lets a foundation-model lab differentiate on the next release, and it gives robotics teams a reason to pair a specialist spatial module with a frontier VLM in the meantime rather than wait for one model to solve it.