ByteDance Reframes VLM Data Mixing as Convex Optimization

Most of what separates a frontier vision-language model from an open one isn't architecture, it's the mixture recipe: which datasets, in what ratios, and how to decide when a new source is worth admitting. That's the part frontier labs don't publish. A team at ByteDance's Commercial AI Team is trying to move the recipe question out of intuition and into something you can actually solve with an optimizer.

Their paper, posted to Hugging Face, calls the framework DecoupleMix. The idea is to split mixture design into two orthogonal problems. Inter-class ratios (roughly, how much OCR versus Video versus General) get a single-variable iterative search. Intra-class composition (which specific datasets inside a category, and in what proportion) becomes a constrained convex optimization over dataset-level Quality and Difficulty scores, with an entropy term to prevent collapse onto a few high-scoring sources. Datasets are scored as whole units by Seed-1.6 acting as judge, and the authors report Spearman correlations of 0.82 for Quality and 0.75 for Difficulty against human ratings.

The result they lead with is transferability. Ratios found on a small proxy carry up: applied without retuning to a 32B model, the recipe beats matched-budget stacking on 13 of 16 benchmarks and lifts the average from 61.5 to 62.7. At the headline scale, a 4B model trained on 80B additional multimodal continue-pretraining tokens ties Qwen3-VL-4B on overall average (62.5 vs 62.5), while the 1B model comes within striking distance of the larger Qwen3-VL-2B (52.3 vs 53.7). Gains concentrate on reasoning-heavy benchmarks (PuzzleVQA 49.0 vs 43.0; CharXiv 63.7 vs 59.9) and trail slightly on perception-heavy ones like OCRBench.

The honest caveat is that the authors themselves flag the Qwen3-VL comparison as descriptive, not causal, because scale and training differ. The controlled evidence is the matched-budget scaling study, where DecoupleMix beats heuristic stacking by +1.6, +2.0 and +1.4 average points at 2.5B, 5B and 10B tokens. What the reporting doesn't give you is any signal on whether the recipe transfers off the Qwen3-VL family, or how much compute the proxy search itself burned. There's also a judge-model dependency: if Seed-1.6's scoring drifts, so does everything downstream.

For teams building VLMs outside the frontier labs, the appeal is that this reframes the expensive part of the job. Instead of stacking every dataset that passes a quality filter and burning tokens to find out, you get a cheap proxy search and an attributable admission test for each new source. That's the kind of engineering discipline the open side of the VLM race has largely been missing.