huggingface.co web signal

Apple-Cornell FastFrames Pushes AR Video to 36ms/Frame on B200

TL;DR

  • FastFrames' 1.3B variant runs at 36 ms per frame (28 RPS) on a single B200 GPU, matching a baseline nearly 8× slower.
  • A ConvNeXt-based pixel head replaces the VAE decoder and cuts decoding latency by 147× while matching visual quality.
  • The 14B variant reaches 45 ms per frame (22 RPS) on an 8× B200 node, under the 50 ms interactive-latency threshold the authors cite.

"A single VAE decoding step takes as long as 3.5 DiT passes." That line from Apple and Cornell's FastFrames paper is the kind of finding that reorders a research agenda. In autoregressive video diffusion, the bottleneck people have been chasing — iterative denoising — is not where the time is going anymore.

The paper profiles Causal Forcing++, the fastest few-step AR video diffusion baseline at the time of writing, and finds a 271 ms-per-frame budget at 480p in which "only 30% of wall time is spent on diffusion sampling." The rest sits in VAE decoding and in a DiT evaluation used solely to refresh the KV cache after each denoised latent.

FastFrames swaps both out. The VAE decoder is replaced with a ConvNeXt-based head that operates at low spatial resolution, followed by a single linear layer that unpatchifies and upsamples the latents to pixel dimensions, "reducing decoding latency by 147× while matching quality." The KV-cache DiT pass gets a lightweight network in its place, cutting that update "10x without measurable degradation in quality."

The headline numbers come straight from the abstract. The 1.3B variant "runs at 36 ms per frame, or 28 responses per second (RPS), on a single B200 GPU while matching the generation quality of a state-of-the-art model that is nearly 8× slower." The 14B variant lands at 45 ms (22 RPS) on an 8× B200 node with "significantly better visual quality and realism." Both come in under the 50 ms delay-detection threshold the authors cite from human-computer interaction research as the bar for seamless interactivity.

The authors release code, weights and a project page, and point at two downstream uses: camera-conditioned and physics-conditioned video generation. It lands in a run of KV-cache and pixel-decode latency work moving through AI video research this month.