Native Multimodal Scaling Laws Hinge on Data Mixture Choice

A new arXiv paper on scaling laws for native multimodal pretraining lands on a finding that reshuffles how teams should think about their training budget for vision-language models. The compute-optimal recipe for models trained from scratch is highly sensitive to the mixture of the training data, in a way that pure-language training is not.

The paper from Haoyuan Wu and colleagues argues that minimum loss follows a predictable compute law, with both model sizes and token counts scaling as power laws (the familiar shape from text-only work). What changes for multimodal is the allocation between parameters and tokens. Language-only allocation stays largely invariant to the composition of the data, while the multimodal allocation law is highly sensitive to data mixture ratios. Text-heavy mixtures, the authors report, only become compute-efficient at larger model scales, which means the right model size for a run depends on the mix you feed it.

Why this matters if you are not training frontier models yourself: it changes the shape of the trade-off. The default frame for the last few years has been add parameters, add tokens. A result like this says that for multimodal, the data recipe is a higher-order lever, and getting it wrong can push a run off the efficiency frontier before a single extra dollar is spent on GPUs. It also gives labs choosing between training natively from scratch and adapting an existing LLM a concrete frontier to plan against.

The honest caveats are the usual ones for a scaling-law abstract. The authors report the shape of the curves; what the writeup does not spell out for a hallway conversation is the specific data ratios, the exact model sizes and token counts on the frontier, or how the native-from-scratch route directly compares to LLM adaptation in dollars and downstream benchmarks. Take the framing as a planning tool, not as settled recipe values.

The upside is real for smaller teams. If mixture is the lever, the expensive class of experimentation to find it is now a paper someone else has written, and the payoff of getting it right shows up in the positive cross-modal transfer, improved pure-text spatial reasoning, and multimodal in-context learning the authors also report.