SenseTime unifies computer vision as multimodal generation

A team from SenseTime Research, with collaborators at Nanyang Technological University and the Chinese University of Hong Kong, has published a paper on Hugging Face arguing that most of computer vision can be expressed as text-plus-image generation from a single unified multimodal model, with no task-specific prediction heads. They call the resulting system SenseNova-Vision, and they train it by adapting an off-the-shelf UMM, Bagel-7B-MoT, on a converted corpus of computer vision annotations.

What makes this more than a repackaging exercise is the scope. One model is asked to produce bounding boxes, OCR strings, keypoints and camera parameters as text, and masks, depth maps, surface normals and point maps as images, across four families the authors group as structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry. The training corpus, SN-VC, is released in part as SN-VC-50M, a 50-million-example subset of generated and curated targets, with the remainder reconstructible from released source lists and templates.

The results are competitive rather than dominant, and the paper is fairly honest about that. On dense depth and surface normal benchmarks SenseNova-Vision sits close to geometry specialists such as MoGe-2 and Depth Anything V2, without beating them everywhere. On multi-view point-map reconstruction and camera pose it lags feed-forward specialists like VGGT and Depth Anything 3 on several metrics, and the authors say so directly. The generalist-versus-generalist comparisons look better: against Youtu-VL, they report 53.7 mAP on COCO detection versus 47.1, and 98.1 vs 90.4 on NYUv2 depth.

The honest caveat is that broadening a UMM this way isn't free. After mixed-task fine-tuning, the model's MMVP score reportedly dropped to 79.0 from Bagel's 83.3, though GenEval text-to-image rose to 0.85 from 0.82. What the paper doesn't quantify is inference throughput against specialists, how well one prompt can compose two task families at once, or how much of SN-VC beyond the 50-million released examples a downstream team could realistically reproduce.

For teams building perception into agent or robotics stacks, the pitch is that one prompt-driven model can stand in for a DETR-plus-SAM-plus-OCR-plus-depth pipeline, and inherit the general multimodal skills of the underlying UMM. Take the specifics as reported rather than settled: this is one paper, released with the model, and how far the formulation scales past the benchmarks in the table is the part worth watching.