Inria's PANORAMA grounds caption phrases to panoptic masks

PANORAMA, a vision-language model from a group at Inria and the Czech Institute of Informatics, Robotics and Cybernetics in Prague, tries to caption both the foreground and the background of an image while grounding every referring phrase to a pixel-level mask. The authors (Sara Pieri, Evangelos Kazakos, Shizhe Chen, Josef Sivic and Cordelia Schmid) frame phrase grounding as mask proposal selection: a segmenter proposes candidate masks, and the model learns to pick which, if any, match each phrase.

Alongside the model the group ships PanoCaps, described on the project repository as 'a human-annotated panoptic grounded captioning benchmark with approximately 99% pixel coverage.' It covers roughly 3,500 images, 34,000 masks and 17,900 unique noun phrases. Training leans on two larger auxiliary sets: 118K recaptioned COCONut-PanCap images and 101K adapted MRSeg referring expressions. The authors claim the system 'achieves the best overall grounding on PanoCaps and matches or exceeds specialized models across several pixel-level grounding tasks.'

Underneath, PANORAMA uses a Qwen3-VL-4B-Instruct backbone paired with SAM 3 segmentation weights, and the team has released training, finetuning and evaluation code. Two researchers we follow passed the repo around on the day it landed.

Models, data and code sit under CC BY-NC 4.0, non-commercial research use only.