BLIP-Live trains proactive streaming video agents on 780K samples
TL;DR
- The team released BLIP-Live-780K, a human-annotated dataset defining proactive response content and timing for streaming video agents.
- A new metric, PAUC-F, combines response timeliness with firing precision so models are not rewarded for excessive firing.
- BLIP-Live-Qwen reports state-of-the-art results across four proactive streaming benchmarks with gains up to 44% over baselines.
Streaming video understanding has a problem the field often skips over: knowing when to speak. A paper on arxiv from Kanchana Ranasinghe, Juan Carlos Niebles, Silvio Savarese, Ran Xu and co-authors rebuilds the task around that choice.
They call the framework BLIP-Live, which they describe as 'unifying data, metrics, harness, and training around a shared formulation of proactivity as effective causal response decisions balancing the two competing objectives of responding vs silence.' The release has three pieces: a human-annotated corpus they call BLIP-Live-780K, a new metric, PAUC-F, that 'combines existing response timeliness metrics with firing precision to reward early useful responses without rewarding excessive firing,' and an inference harness paired with training pipelines for vision language models.
Their best model, BLIP-Live-Qwen, is reported to achieve 'state-of-the-art performance across four proactive streaming benchmarks, with gains up to 44% over baselines and outperforming multiple closed API models.' The abstract does not name the closed APIs tested or publish per-benchmark numbers. Code and data 'will be released publicly.'
Originally reported by paper
Read the original article →Original headline: BLIP-Live: 780K-Sample Framework Trains Proactive Streaming Video Agents, Beats Closed APIs on Four Benchmarks