RefCaptioner grounds video captions in multi-image references

Video captioners have gotten fluent at describing what a scene looks like, but they usually fall back to generic nouns rather than naming the specific person, product, or character on screen. A new arXiv paper proposes filling that gap with a task the authors call multi-reference image-grounded video captioning, where the model is given a video plus a set of reference images and has to produce captions whose phrases are actually bound to those references.

The system, RefCaptioner, is trained with what the authors describe as mixed-data SFT combined with a variant of GRPO they call Hierarchical Coverage-Discounted. The training corpus is 20,000 videos paired with 171,354 reference images, and the paper ships a companion benchmark, MRVBench, for evaluating both factuality and grounding across authentic and AI-generated video. The authors' own summary of what the training buys you is fairly specific: gains on reference selection, phrase-level binding, distractor rejection, and cross-reference consistency, with human evaluators reportedly preferring RefCaptioner's captions.

Why this matters if you do not ship video models yourself: this is the direction that turns captioners from generic scene describers into something usable in sports broadcasting, product demos, or personalized video assistants, where saying 'this player' instead of 'a person' is the whole point. Accessibility is the other obvious beneficiary, because a named-entity caption is dramatically more useful to a blind user than a generic one.

The honest caveat is that this is a single paper introducing its own benchmark, and the strongest headline is a human-preference win rather than a hard accuracy number a reader can compare across systems. The abstract does not give head-to-head numbers against closed frontier captioners, and there is no external replication yet. There is also a real dual-use worry that better reference-image grounding makes it easier to identify people in video without consent, which the abstract does not appear to address.

Still, the interesting shift is treating who and what as a first-class captioning task rather than a downstream retrieval hack, and MRVBench is the kind of dataset that could get pulled into other groups' evaluations quickly if the released code lives up to the claim.