DreamFree Skips Video Generation for 11Hz Robot Control

DreamFree runs closed-loop robot control at 11Hz on a 14B video diffusion backbone, and does so without generating any future video.

The design, described in a new arxiv paper, repurposes the diffusion model as a predictive visual encoder rather than letting it imagine next frames. 'Our key idea is to make this predictive knowledge available through visual representations computed from the current observation, enabling action prediction without generating future videos,' the authors write. The visual encoder is trained to predict a frozen video diffusion model's representation from the current frame, and an action module reads action-relevant information off it.

The problem the design targets is stated plainly: 'iterative video denoising is costly, and inaccurate visual predictions can compromise action prediction.'

The paper claims 'higher success rates than state-of-the-art VLAs and WAMs' across real-world and simulated tasks. The abstract names no baselines and publishes no per-task numbers. It also claims the method can learn new robot tasks from video demonstrations, including 'videos from a different embodiment, without action annotation for the new tasks.'