SaMer cuts 93% of visual tokens, lifts ColPali retrieval

A quiet result out of a new Hugging Face-listed paper argues that most of the visual tokens a multi-vector image retriever stores are doing very little work, and that if you throw the right ones out, retrieval actually gets better, not worse. The method is called SaMer, short for Semantic-aware Merging, and the code is on GitHub.

The setup is worth explaining. Systems like ColPali and ColQwen2 do image search by keeping hundreds to over a thousand token embeddings per image, then comparing every query token against every image token at scoring time. That late-interaction step is what lets a query about a specific object or attribute latch onto a specific patch, but the cost is real: storage grows with every image, and scoring cost scales with the token count. The authors' claim is that a small set of merged tokens can carry the same evidence if you merge them the right way.

Their method compresses image-side post-projector tokens into K representative centroids, and at K=64 they report removing more than 93% of image-side tokens and shrinking ColPali storage by 16.09 times. The counterintuitive part is the accuracy line. Recall@1 on Flickr30K reportedly rises from 77.0 to 82.4 for ColPali and from 73.6 to 79.3 for ColQwen2, and MSCOCO Recall@1 moves from 47.4 to 51.6 with ColPali. The trick they credit is using object annotations during training as a merge prior that discourages tokens from different object instances from collapsing into the same centroid. At inference time, no bounding boxes or detectors are required, and only the shared projection layer is adapted; the vision encoder and language backbone stay frozen.

The honest caveat is that these are retriever-only benchmark numbers on a single paper, adapted on the Flickr30K-Entities training split, and the gains do not extend cleanly to DocVQA, where the authors themselves note that aggressive token compression can strip sparse OCR and layout evidence rather than redundant visual patches. The comparisons are against pruning- and pooling-style baselines like H-Pool, HPC, and SAP rather than production-tuned quantization stacks, and the paper reports storage compression rather than wall-clock latency.

Still, the direction is the interesting part. If holding onto object-level evidence rather than raw patch count is what matters for late-interaction retrieval, teams shipping visual RAG or multimodal search have a plausible path to shrinking multi-vector indexes by an order of magnitude without giving up recall, so long as the numbers hold outside the paper's chosen benchmarks.