FVAttn Load-Balances Video DiT for 4.41× Attention Speedup

A new arXiv preprint from Hao Liu and colleagues makes a small but pointed argument about why multi-GPU inference for video diffusion transformers is slower than the attention math suggests: the sparse attention people bolt on to fix cost is itself creating the bottleneck.

The setup is familiar for anyone doing high-resolution video generation. Self-attention dominates cost, so teams add training-free sparse attention with adaptive Top-p routing. Once that is spread across GPUs with sequence parallelism, each rank ends up with a different amount of work, and the whole step waits on the slowest one. The authors call this a rank-level straggler problem, meaning the theoretical savings from sparsity get eaten by idle time on the faster ranks.

Their system, FVAttn, keeps the Top-p routing plus a Top-k safety floor and video-aware block organization, but then repairs the materialized mask at runtime. Heavy attention heads are migrated to lighter ranks over peer-to-peer communication to shorten the critical path, and residual slack on non-critical ranks is refilled with additional high-value blocks, with the scheduling overhead overlapped behind existing compute. On a step-distilled Wan2.2 I2V model, they report average load imbalance dropping from 1.34 to 1.08, a 4.41× attention speedup over FlashAttention, and a 2.02–2.11× DiT inference speedup at what the paper describes as competitive video quality.

The honest caveat is that these are single-paper preprint numbers on one model family, and 'competitive video quality' is the abstract's phrase rather than a quantified human-preference or FVD delta a serving team can audit. What the write-up does not give you, at least in the abstract, is the exact GPU count or interconnect the benchmarks ran on, which matters a lot for whether the P2P migration cost holds up outside a favorable test bed.

If the results replicate, the more interesting shift is where the optimization surface moves. Sparse attention libraries have spent the last stretch fighting for better masks; this argues the next win sits on the scheduler above them, and the inference-infra teams shipping video at scale, plus the runtime vendors selling to them, are the ones who would feel it first.