PANORAMA team releases PanoCaps, 3,470-image grounding benchmark

The Panorama-grounding team on Hugging Face has published three datasets tied to their arXiv paper on panoptic grounded captioning. PanoCaps, the headline benchmark, pairs 3,470 human-annotated images with 34K panoptic regions and more than 99% region coverage per scene, split 2,070 train / 420 val / 980 test.

The paper, from Sara Pieri, Evangelos Kazakos, Shizhe Chen, Josef Sivic and Cordelia Schmid at the Czech Technical University in Prague and Inria / École normale supérieure, frames the task as 'generating dense captions that describe both foreground objects and background regions while grounding each referring phrase with pixel-level masks.' Their PANORAMA model layers a Qwen3-VL backbone (2B or 4B variants) on top of SAM 3, bridging the two with a 256-dimensional concept vector that conditions phrase-specific mask proposals.

PANORAMA-4B scores 68.0 CAPTURE, 61.2 AP50 and 45.6 gPQ on the PanoCaps test set; the authors describe the result as 'best overall performance on PanoCaps and competitive results across established grounding benchmarks.'

PanoCaps carries a CC-BY-NC-4.0 license, with underlying images curated from ADE20K, COCONut and VIPSeg. Two researchers from our directory flagged the release the same day.