Fudan SAM-MT holds 36+ FPS across 10-target video segmentation

A paper out of Fudan University and Shanghai University of Finance and Economics, posted to Hugging Face, points at a real bottleneck in the current generation of video segmentation models: they scale badly the moment you ask them to track more than one thing at once. SAM2, the current default, sees its frame rate collapse from 37.2 FPS on a single target to 17.8 FPS on three and 12.4 FPS on five, because the pipeline is essentially replicated per object. SAM-MT, the authors' extension, holds above 36 FPS at 10 targets and reports the same near-flat trend out to 20.

The trick is architectural. Instead of running the SAM2 loop independently for each target, SAM-MT represents each user-specified object as a lightweight target query and shares a single global-context representation across all of them. A decoupled masked attention scheme lets those queries reach the global context but blocks them from leaking into each other, and a query-based sparse memory replaces SAM2's dense per-target memory, which the paper says typically runs 4096 tokens per object, with a single query token per object per frame. The model is initialized from a SAM2.1-B+ checkpoint and trained on 8 NVIDIA A6000 GPUs, on a filtered SA-V subset restricted to sequences with at least three concurrent targets.

On the six VOS benchmarks the paper reports, SAM-MT is competitive rather than dominant. It reaches 43.0 on MOSEv2's J&F metric, edging SAM2.1-B+ at 41.1 and roughly matching Cutie-base at 42.8, with slightly larger gains on long-term splits (2.0 and 2.3 points over SAM2.1-B+ on LVOSv2 and LVOSv1). VRAM tells the sharper story: SAM-MT climbs from about 3094 MB to 3785 MB going from 1 to 20 targets, while SAM2.1-B+ balloons from 3043 MB to 8585 MB on the same test. For anyone thinking about deploying video segmentation into crowded driving or robotics scenes, that memory curve is the number that matters most.

The honest caveats are the ones the paper itself flags. The scalability benchmark is synthetic, 20 sequences the authors constructed and padded or truncated to 100 frames, because existing VOS datasets are still dominated by single-target clips (79% of MOSEv2-val, per the paper). And SAM-MT inherits SAM's vision-only architecture, which the authors acknowledge lacks the reasoning needed for higher-level tasks. What the reporting doesn't give you is any hosted API, latency numbers on cheaper hardware than the A6000, or a head-to-head against commercial closed trackers. Still, if you build tools where the target count is genuinely variable, this is the design paper worth reading closely.