huggingface.co web signal

Hugging Face trains ColBERT to retain 97.8% at 4x pooling

By Published Updated

TL;DR

  • The ordinary ModernBERT multi-vector model retains 94.6% of its unpooled NanoBEIR nDCG@10 at pool_factor=4; pooling-aware training raises that to 97.8%.
  • Training applies the retrieval loss at multiple compression levels, with Ward hierarchical clustering merging similar vectors and a straight-through estimator carrying gradients back.
  • Gains widen under harsh compression: at max_tokens=4 the pooling-aware model scores 43.00 nDCG@10 versus 37.22 for the baseline, +5.78 points.

At `pool_factor=4`, the ordinary multi-vector model keeps 94.6% of its retrieval quality. Teach the same encoder to anticipate the pooling during training, and it keeps 97.8% at the same quarter-sized storage.

That is the headline result from a Hugging Face blog post dated November 11, 2026, by Tom Aarsen and collaborators. Two of the retrieval researchers we track surfaced it the same week. The method bolts a pooled retrieval term onto the training objective, with Ward hierarchical clustering merging similar token vectors and a straight-through estimator pushing gradients back through the merge. Training runs one epoch over 120,000 MS MARCO triplets starting from ModernBERT-base. 'With a few extra lines in the training objective, a smaller document representation becomes something the model actively learns to use,' Aarsen writes.

The gain widens as compression hardens. At `max_tokens=4`, four vectors per document, the pooling-aware model scores 43.00 nDCG@10 against the baseline's 37.22, a +5.78 jump. A dedicated CUDA kernel for Ward clustering, the post reports, runs 30 to 150 times faster than SciPy. Evaluation is on NanoBEIR@13 only, with a single seed.

Shared on Bluesky by 2 AI experts

Originally reported by huggingface.co

Read the original article →

Original headline: Token Pooling-Aware Training for Multi-Vector Embedding Models