CMU team unifies foundation-model training into one algorithm

A new arxiv preprint from a Carnegie Mellon-led team argues that the staged pipeline producing modern foundation models (pretraining, midtraining, supervised fine-tuning, reinforcement learning) is a convention, not a mathematical necessity.

"Foundation models are typically trained in stages: pretraining, midtraining, supervised fine-tuning (SFT), and reinforcement learning (RL), with changes in both data and learning algorithms," the paper's abstract reads. "We show that this algorithmic separation is not fundamental." The construction, as the authors describe it: by "deriving suitable RL objectives as maximum-likelihood fitting to model-generated target distributions," they place all four stages under one likelihood-based trainer they call unified training, in which static data and RL-generated completions share a common weighted likelihood loss. The foundation-model training lifecycle, the paper says, becomes "a problem of choosing which data to train on, in what proportions, and when."

The experiments are small. A 124M-parameter model trained from scratch on a 20B-token pretraining corpus and a 510M-token midtraining corpus is used for controlled reasoning puzzles. A 0.76B-parameter model is evaluated on MATH-500, GSM8K, HumanEval+ and MBPP+ with a 240M-token heterogeneous midtraining corpus. The paper reports that unified training achieves "higher accuracy and coverage on reasoning tasks compared to staged training within equivalent compute budgets," that mixing midtraining data during pretraining substantially reduces forgetting of general capabilities, and that unified-trained models serve as better starting points for a subsequent coding-RL phase after math training.

The paper lists Carnegie Mellon University as the primary affiliation, with Johns Hopkins University and Aurora Innovation also contributing, and support from NSF, ONR, a Qualcomm Fellowship, and compute from OLCF and ALCF. The authors are Yuda Song, Alvin Zhang, Fahim Tajwar, Daman Arora, Jeff Schneider, Daniel Khashabi, Ruslan Salakhutdinov, J. Andrew Bagnell, Aarti Singh, and Andrea Zanette. It was posted on October 29, 2026. The largest run reported is 0.76B parameters, well below frontier scale.