Zero-WAM Hits 47% on Unseen Robot Tasks From Human Videos

A causal video-action model scored 46.95% average zero-shot success on seven unseen tasks in the RoboTwin 2.0 simulator by watching a human do the task first, according to a paper posted to arXiv by Jiaming Zhou and collaborators.

The gap over the strongest video-action baseline, LingBot-VA, was 29.50 percentage points.

The method, called Zero-WAM, borrows in-context learning from large language models: rather than fine-tune per task, the policy consumes a human video at deployment time and autoregressively predicts future robot observations plus executable actions. To feed it, the authors built a pipeline they call HumanGen, which "converts task-sampled robot trajectories into semantically matched human video instructions, yielding 74.2K human-robot ICL pairs over 8.6K tasks."

The training objective is designed against a specific failure mode. The paper describes "an in-context future chunk prediction objective to reduce shortcut learning and strengthen the use of human video prompts," the worry being that a model exposed to enough demonstrations will start pattern-matching on training tasks rather than reading the prompt in front of it.

Real-world results are reported qualitatively. The paper says Zero-WAM "follows human video guidance to generalize to unseen task configurations involving multi-object scenes, long-horizon manipulation, and fine-grained insertion," without publishing per-task success rates for those runs. The seven-task simulation number is the only headline figure.