HKUST, Tencent FAPL Trains Video DiTs to Stop Bypassing MLLM Latents

Video generators that advertise multimodal LLM conditioning often don't actually use it. That is the central empirical claim of a new HKUST and Tencent Video paper, which finds that when researchers wire an MLLM into a Video Diffusion Transformer, "a Video Diffusion Transformer (DiT) may bypass them through native conditioning pathways," reducing training loss by leaning on text-encoder and VAE-reference inputs instead.

The problem gets concrete in the paper's Figure 2. Under MLLM-only conditioning, "Omni-Video 2 (1.3B DiT) produces noise, while Omni-Video 2 (14B DiT) and UniVideo produce artifacts and depict one person's successive actions as two people acting differently." Under conflicting conditions, the authors report, VAE references dominate the output. The MLLM features, in other words, are decorative.

Zhe Wang and Anyi Rao of HKUST's MMLab, with eight co-authors from Tencent Video and École Polytechnique, propose Factor-Aware Predictive Learning (FAPL) on a 13B UniVideo backbone. Stage I freezes the MLLM and connector and trains the DiT with the projected MLLM latents as its only content condition, so the generator cannot fall back on text and visual-reference pathways. Stage II uses paired videos that differ along only one of four cinematic factors (character, scene, action, or camera), swaps the matching span inside the projected latent sequence, and supervises the DiT with the paired target video. Stage III then freezes the trained DiT and fine-tunes the MLLM and connector so text-only and text-plus-complete-video inputs share one video target.

Training runs 10K, 15K, and 10K steps across the three stages on 100K quality-filtered source-target video pairs, 25K per factor, drawn from CoinVE-200K and OmniVBench's released training data. The authors use AdamW at a 1e-5 learning rate with BF16 and gradient checkpointing; inference uses 50 sampling steps and classifier-free guidance of 5.0. Evaluation runs on a held-out CineFactor-10K test set covering sparse-to-video, factor-aware editing, and multi-shot generation.

The abstract reports "consistently better performance on three cinematic video generation tasks" against the original UniVideo and the Omni-Video 2 1.3B and 14B baselines. The retrieved text publishes no per-task numbers against those baselines. The paper landed on arXiv on 23 October 2026, part of a run of cinematic-video work hitting our tracker this week alongside Peking University and Kling's Verus-Insight artifact detector.