DataPrep-Bench scores LLMs as training data preparators

Training data has been the black box behind every model release. A new benchmark out of Peking University and collaborators, posted to arXiv as DataPrep-Bench, tries to open it up by asking a specific question: how well do LLMs, agents, and data-centric workflows actually prepare the training data that other LLMs then learn from?

The setup is two-sided. One track measures data construction, whether an LLM can turn raw sources into supervised training data. The other measures data quality evaluation, whether an LLM can predict how useful a candidate dataset will be before you spend the compute training on it. Both tracks are scored on downstream results across six domains: general text, mathematical reasoning, science knowledge and reasoning, medical, financial, and legal. The base models under test include Qwen2.5-7B, Llama-3.1-8B, and, on the quality-evaluation side, Mistral-7B-v0.3.

The headline number the authors report is roughly 20 points. Their skill-guided agentic approach, which they call Data-Construction-Skill, "lifts the Dolly-only baseline by nearly 20 points absolute on Llama-3.1-8B Finance." Their proposed quality metric, DAS (Distributional Alignment Score), is described as "the only metric clearing r>0.70 simultaneously in Math, Science, and Medical" and as attaining "the strongest cross-model correlation in four of six domains."

Why this is worth watching if you don't personally train models: the paper's argument is that dataset preparation is now a discrete capability you can grade, and one that today's off-the-shelf LLMs are meaningfully worse at than a skill-guided pipeline. If the result holds, it reframes some of the "we need more data" story into "we need better data preparators," which is a very different budget line.

The honest caveats are the usual ones for a first-of-its-kind benchmark. The 20-point figure is a single-domain, single-base-model result rather than a sweep across all six domains, and this is a preprint with no independent replication yet. What the paper doesn't give you is how these preparator agents behave on truly proprietary raw corpora, or how much of the gain survives when you swap the base model or push past the 7B–8B range. The direction is still the interesting part: data preparation is being treated as a first-class model-training skill rather than plumbing.