Falcon cuts audio-visual speaker-extraction compute 20.2×
TL;DR
- Falcon reduces computation 20.2× and GPU inference latency 11.1× versus the causal AV-TFGridNet baseline, while cutting parameters by 91.9%.
- The authors release RealSSA, a new AVTSE benchmark with 1–5 dynamic near-field interferers and 10–15 far-field speakers in harder scenes.
- An encoder-space speaker contrastive loss suppresses near-field leakage at zero inference cost and transfers across backbones.
Falcon, a new streaming audio-visual target speaker extraction model, cuts computation 20.2 times and GPU inference latency 11.1 times versus the causal AV-TFGridNet baseline while shedding 91.9% of its parameters, according to a preprint on arXiv by Wendi Sang, Kai Li and co-authors at Tsinghua.
The authors argue causal AVTSE has been measured on tasks that are too easy. "Standard benchmarks fix the number of speakers at two and ignore the near-/far-field interference, reverberation, and sudden noise of real scenes," the paper states. Their proposed fix is RealSSA, a two-part benchmark pairing simulated scenes with 1 to 5 near-field interferers (and 10 to 15 far-field voices in the harder cut) with a held-out set of real recordings.
Falcon's separator does multi-scale time-frequency separation in a single encoder-decoder pass, trained with an encoder-space speaker contrastive loss that the paper says "suppresses near-field leakage at zero inference cost and transfers across backbones." The abstract claims state-of-the-art extraction on RealSSA-Sim, LRS2-2Mix, LRS3-2Mix and VoxCeleb2-2Mix, though it publishes no per-benchmark scores.
Originally reported by paper
Read the original article →Original headline: Falcon AVTSE Cuts Speaker-Extraction Compute 20× With New Real-World Benchmark