Dwarkesh Study: Data Beat Model Recipes 3.24x in Pretraining

Dwarkesh Patel and Jerry Han report that between 2019 and 2025, data improvements delivered roughly 3.24 times more compute-efficiency gains in pretraining than model-recipe changes did, measured at a 1e19 FLOPs training budget. In a post on Dwarkesh's site, the pair break the number down as "12.0x for data and 3.7x for models."

They trained model recipes from GPT-2 through OLMo-2 on corpuses ranging from OpenWebText to UltraFineWeb, at multiple compute budgets and with several random seeds, then scored the runs on OLMES, "which aggregates 10 different relatively easy benchmarks, mostly multiple choice QA." The two axes look largely independent: "88% of the variance in the OLMES score can be explained by additive effects of the model and data improvements."

Their headline annual rate, 1.57x year-over-year with a 95% confidence interval of 1.49 to 1.65, lands well below prior estimates. "This is indeed much lower than Anson Ho et al.'s mean estimate of 3x YoY," they write, pointing to Ho's own suggestion that "most software progress might actually be due to data quality improvements."

The authors flag their own limits inline. "Even our experiment was done at an extremely small scale," they note, arguing scale-dependent innovations and inference-side gains would not show up in models this small. On why data may dominate at these sizes, they write that small models "don't have that much capacity, and so you have to be really careful about what you stuff into them."

They also decline to overload the finding. "Whether pretraining progress in isolation will speed up or slow down is not really the most important question for overall AI progress," they write, "because so many of the gains over the last two years have come from RL." That maps to post-training work like this week's NeoHorse-1 paper, which frames recursive self-improvement as a routing-harness loop. The closing worry is about the input side: with data quality doing most of the work, they ask whether synthetic data can keep filling the tank, framing it as "are we running out of cargo?"