KAIST DreamX Repurposes Video Diffusion as Latent-Action Learner
TL;DR
- KAIST AI's DreamX compresses each latent video frame 300x, from 1,200 tokens down to 4, by reusing a pretrained video diffusion model.
- On LARY-Bench, the paper claims state-of-the-art performance among latent action models and reports zero-shot action transfer across human and robot embodiments.
- The authors pitch the compact latent space as a way to cut the compute cost of video-diffusion-based robot policies.
Researchers at KAIST AI have taken a pretrained video diffusion model and treated it as a source of robot actions, compressing each latent video frame by 300×, from 1,200 tokens down to 4, in a system they call DreamX.
The paper argues that the diffusion model's generative prior lets the team pull "action-relevant dynamics disentangled from visual appearance" out of the video model's spatiotemporal features. The goal is a single action vocabulary that survives changes in embodiment, environment, and action parameterization. "The resulting latent space serves as a shared action interface across embodiments (human and robot), environments, and action parameterizations," the authors write.
On LARY-Bench, DreamX "achieves state-of-the-art performance among latent action models," according to the abstract. The paper publishes no per-task success numbers and names no specific video diffusion backbone, so what is on offer here is the compression ratio and the transfer claim, not a raw benchmark score.
The group also reports that actions encoded in one setting remain effective in another, supporting what they describe as "any-to-any video transfer and zero-shot action transfer across diverse embodiments and environments." It lands alongside another KAIST robotics paper filed in our tracker earlier today.
Originally reported by huggingface.co
Read the original article →Original headline: DreamX Repurposes a Video Diffusion Model Into a Universal Latent-Action Learner, Compresses 1,200 Frames to 4 Tokens