PANORAMA grounds every caption phrase to pixel-level masks

A new vision-language model from researchers at Inria/ENS and the Czech Technical University in Prague, called PANORAMA, generates full-scene captions in which every phrase — foreground objects and background regions alike — is tied to a pixel-level mask, and its 4B-parameter version outscores both specialized grounding systems and far larger generalist models on the authors' own benchmark.

In the paper hosted on Hugging Face, Evangelos Kazakos, Shizhe Chen, Josef Sivic and Cordelia Schmid formulate phrase grounding as "selection from a phrase-conditioned pool of mask proposals," conditioning a pretrained SAM 3 segmenter on contextual phrase representations and training it jointly with caption generation. On the PanoCaps test set, PANORAMA-4B reaches a generalized Panoptic Quality score of 45.6, ahead of SAMTok-8B at 44.5 and GLaMM-7B at 30.4, and well above the much larger generalist Qwen3-VL-235B at 26.1 and Gemini 2.5 Pro at 22.7. The authors describe this as "the best overall grounding on PanoCaps."

The release ships a benchmark alongside the method. PanoCaps contains 3,470 human-annotated images with "near-complete pixel coverage" of about 99%, split into 2,070 train, 420 validation and 980 test images, and holding 34,100 masks across 31,300 grounded entities. In a training-data ablation, the authors argue that quality matters more than raw scale: PanoCaps beats COCONut-PanCap "despite 57x fewer training samples" (gPQ 44.0 vs 37.9), which they frame as evidence that "annotation quality and alignment fidelity can substantially offset dataset scale." Two researchers we track posted the link the same week it went up.

Only the datasets — PanoCaps, COCONut-PanCap-Recaptioned and MRSeg-Referring-Expressions — are currently published on the group's Hugging Face organization page. No PANORAMA model weights are listed there yet.