Alibaba DAMO's RynnWorld-Latent Pretrains on 553.5M Frames

The pretraining corpus runs to roughly 553.5 million frames: about 6,190 hours of video across 23 datasets and 2.57 million episodes, mixing human interaction clips, robot demonstrations and simulation rollouts under one conditioning interface.

That interface is what the paper actually proposes. "This representation enables joint pretraining on human interaction videos, robot demonstrations, and simulation rollouts without requiring a common native action space," the Alibaba DAMO Academy authors write, referring to the 3D-grounded latent actions produced by a frozen RynnLAM encoder. The world model learns to predict future video latents from those shared action tokens rather than from embodiment-specific joint commands.

The team also used the stack to synthesize training data. A corpus they call RynnWorld-Latent-80M, drawn from EPIC-KITCHENS, EgoDex and Something-Something, holds about 84.8 million generated frames (roughly 800 hours) of pseudo-robot rollouts, built by transferring source motion onto an edited initial frame and pairing it with retargeted actions.

On downstream policy benchmarks, synthetic mid-training moved π₀ on LIBERO-Plus from 53.6% to 79.4%, and π₀.₅ Plate-Placement from 82.86% to 91.43%.

The paper evaluates on two real-robot platforms, Marvin-WUJI (dexterous hands) and Astribot-S1 (mobile gripper), running 35 trials per task across six manipulation tasks. The abstract reports "improved video prediction metrics on public benchmarks and two real-robot platforms, alongside qualitative action following when different action sequences share the same initial observation." Per-detector numbers for the real-robot comparisons are tabulated but the abstract itself gives no head-to-head baseline summary.