CMU-Hopkins Paper Folds Pretrain, SFT, RL Into One Loss

A new paper from Carnegie Mellon, Johns Hopkins and Aurora Innovation argues that pretraining, midtraining, supervised fine-tuning and reinforcement learning do not need to be separate algorithms at all. The authors derive RL objectives as maximum-likelihood fits to model-generated "success-conditioned" distributions, which lets them run every stage through one weighted negative-log-likelihood loss on a time-varying mixture of data sources. They call the recipe unified training, and in the abstract on Hugging Face they write that "this algorithmic separation is not fundamental."

The empirical work is deliberately small. One study trains a 124M-parameter model on 20B pretraining tokens of ClimbMix and Dolmino plus a 510M-token oracle-solution midtraining corpus, then runs 6,000 or 12,000 RL updates on logic puzzles. A second study scales to 0.76B parameters and reports results on MATH-500, GSM8K, HumanEval+, MBPP+ and SciQ. The authors say unified training "improves the accuracy–compute tradeoff and task coverage while preserving broader capabilities, and provides a stronger starting point for continual learning."

The paper is explicit about what it does not do. It studies one RL task at a time, offers no automated method for selecting mixture weights, and does not establish scaling laws across model sizes or data budgets. It arrives into a crowded week of training-method work: AI Weekly has tracked 62 fine-tuning stories in the last 90 days, including Shanghai AI Lab's long-CoT critic result from October 27.