DataComp-VLM: Data Mixing Beats Filtering for Open VLMs

For the last couple of years the recipe for better vision-language models has read like a filtering exercise, throw a bigger web dump at CLIP-score filters and text-quality classifiers and see what survives. A new open benchmark on arXiv makes a different claim worth sitting with, that what actually moves the needle at scale is the mix, not the filter.

The setup is DataComp-VLM (DCVLM), positioned by its authors as a benchmark for controlled data-centric experiments to improve VLM training. They assembled 160 datasets across four data types (image-caption pairs, multimodal interleaved documents, text-only, and instruction-tuning data) into a corpus of 6T multimodal tokens. Submissions can test curation strategies across 1B to 8B models and 6.25B to 200B token budgets, evaluated on up to 52 downstream benchmarks across 9 domains. That is a lot of dials, and the point of the benchmark is that you can turn one at a time.

The headline finding from their own experiments is that data mixing, not filtering, is key to a high-quality training dataset, and that instruction-heavy mixtures scale better than caption-heavy ones, with gains widening at larger scales. If that holds up under other groups' replications, it reorders a lot of the received wisdom about VLM pretraining and pulls research effort away from ever-fancier filters toward the more prosaic question of what proportions of what to feed the model.

The honest caveat is that this is one research group's finding on their own newly published benchmark, and benchmark-topping mixtures do not always translate cleanly to production models with different architectures, alignments, and safety layers. What the reporting here doesn't give you is a marginal cost story, how much compute you save or spend hunting the right mixture versus running one more round of filtering, or a picture of how these mixtures behave once instruction tuning and preference optimization happen downstream.

The upside, and it is a real one, is for academic and open-source labs. A shared, reproducible testbed at meaningful scales lets small teams answer real training-data questions without paying flagship-scale compute bills, and it gives the community a common yardstick for the next round of arguments about what actually makes a good VLM.