GigaWorld-1 debuts WMBench for robot policy evaluation

For anyone building robot policies, the expensive part is not the training, it is the evaluation. Every candidate has to be rolled out on real hardware, under human supervision, for hours. A new arXiv paper from a group calling itself the GigaWorld Team proposes that a video world model can stand in for a lot of that physical testing, and it releases both the model and a benchmark, WMBench, to make the case.

The study pairs more than 324,000 simulated rollouts against actual robot executions across seven video world models and four action-encoding schemes. Training draws on 12,980 hours of video, mixing internet clips, physics simulations, open-source robot datasets, human hand data, and proprietary demonstrations. The largest release, GigaWorld-1-Plus at 5 billion parameters, posts a benchmark score of 0.6834; a 1.3-billion parameter Nano variant reaches 0.6716 and is reported to outperform larger general-purpose models.

The counterintuitive claim is the part worth chewing on. The team reports that evaluator quality is driven less by short-term visual realism and more by long-horizon, action-faithful rollout consistency. Translated, making a world model look photoreal frame to frame matters less than making it act like the robot would act all the way through a task. That flips the usual instinct in generative video, where perceptual sharpness gets treated as the win condition. The project page also reports a VLM-based evaluator that hits 87.80 percent exact agreement with human raters across more than 5,000 videos, and rollouts that hold together past 11,000 frames on a single RTX 4090.

The honest caveat is that a benchmark authored by the same team releasing the winning model on it is not a neutral yardstick, and the write-up does not walk through how WMBench transfers beyond manipulation to locomotion or navigation. Nearly one in eight video judgments where the VLM and a human disagree is not a small margin when policy selection is on the line. Take the specifics as reported, not as a settled comparison against the closed labs, which are not in the seven-model sweep.

If this holds up under outside replication, the upside is concrete. Small robotics groups without teleoperation fleets get an open evaluation surface that actually runs on consumer hardware, and the broader field gets a common way to argue about whether a policy is any good before it ever touches a real robot.