CADRE autoencoders rethink compression for video world models

Interactive video world models mostly reuse autoencoders built for a different job: offline text-to-video generation, where no agent is waiting for a frame. A new preprint on arxiv argues this choice sets both the cost per frame and "how long each action waits to take effect," and is often made without the option to retrain at a different compression because the original training data and code are not public.

The authors (Adrien Ramanana Rahary, Yutong Bai, Nicolas Dufour, Patrick Pérez, and David Picard) introduce CADRE, which they describe as "an open family of causal diffusion representation autoencoders spanning multiple spatio-temporal compressions." The members share one diffusion decoder "self-distilled to one or a few steps" and differ only in a small bottleneck on frozen features. They train the same world model on each autoencoder across two domains, manipulation and navigation, with two environments each.

The headline result in the paper's own words: "CADRE forms most of the latency-quality frontier, especially with a moving camera."

Four findings follow. The one the paper labels surprising: "with a still camera, coarser spatial tokens hurt motion far more than image quality." The others. At strong compression, a diffusion decoder is sharper than perceptual regression. In robot manipulation, temporal compression is more effective in the autoencoder than in the world model. And manipulation and navigation call for opposite compressions, temporal when the camera is still, spatial when it moves.

The abstract reports no absolute latency numbers and names no baseline autoencoders.