NVIDIA, Bar-Ilan Post RecCAR to Close Video Attention Gap

Joint video-audio and video-motion diffusion transformers are, in the authors' words, "architecturally bidirectional but functionally asymmetric." Companion modalities learn to attend to video, but the reciprocal pathway back into video stays weak. That gap is the subject of a new preprint from NVIDIA's Dvir Samuel with Idan Schwartz and Gal Chechik of Bar-Ilan University, posted on Hugging Face.

Their fix is a KL regularizer they call RecCAR, for Reciprocal Cross-modal Attention Regularization. It re-normalizes both cross-attention directions over the video tokens, treats the stronger video-to-modality map as a stop-gradient reference, and pulls the weaker direction toward it. The pretrained backbone stays frozen; only LoRA parameters of rank 128 are trained, at loss weight 0.01, for roughly 48 GPU-hours on 4 H100s.

The numbers: applied to EchoMotion, RecCAR lifts the VBench Human Anatomy score from 0.69 to 0.75. Applied to LTX-2 on the 500-prompt T2AV-Compass benchmark, absolute audio-video desynchronization falls from 0.804 to 0.752. The authors state the takeaway plainly: "bidirectional connectivity does not guarantee bidirectional information flow."

The ablation is the load-bearing part. Standard LoRA fine-tuning on the same curated data — 4,292 video-motion pairs and roughly 4,300 VGGSound clips — does not close the gap; on video-motion it degrades N-MPJPE, MPJPE and Human Anatomy. Only the alignment loss produces the gain.

The paper reports no human-preference study alongside its automated scores, and evaluation is confined to VBench, T2AV-Compass and AVGen-Bench. It lands in a crowded field: joint audio-video generation has been one of the more active research subfields on our multimodal tracker this quarter.