V-RAE tops video VAEs on Kinetics-600, converges 6x faster

A new arXiv preprint argues that the way the field builds video latent spaces is backwards: they are tuned for pixel reconstruction, when what matters for generation is semantic structure. V-RAE, from Minghui Guo, Shengqiong Wu and Hao Fei, builds its latents on top of frozen vision foundation model features instead, adding a lightweight temporal pooling module and a video decoder that reconstructs motion from the compressed representation.

The headline numbers are the authors' own. V-RAE reports 2.13 rFVD on Kinetics-600, which the abstract says beats "all evaluated large-scale pretrained video VAEs." Under matched generation settings, the best variant lands gFVD scores of 117.86 on UCF101 and 19.16 on Kinetics-600 "while converging up to 6x faster." The paper also says V-RAE improves future video prediction on Cityscapes over the Wan 2.2 VAE latent space.

The more provocative contribution is a new metric. The authors argue that "reconstruction quality alone is insufficient to characterize generative utility" and introduce tFVD, a temporal-coherence diagnostic that they say "correlates more reliably with downstream generation quality." If that claim survives replication, it is a shot at the rFVD-first evaluation habit that has governed video tokenizer design.

The abstract does not name the four frozen encoders it tests, does not publish per-encoder tables, and there is no third-party confirmation of the speed-up figure yet.