Tsinghua's Puro-2B pretrains on RTX 5090s for under $6.9K

The paper's headline number is deadpan: less than $6.9K. That is the compute cost the authors report for training Puro-2B, a from-scratch language model that "approaches Qwen2.5-1.5B performance under our evaluation protocol," run entirely on consumer-grade RTX 5090 GPUs.

The abstract states the goal plainly: "Language model pretraining has become almost synonymous with prohibitive cost, placing it out of reach for much of the academic and open-source communities." The team, from Tsinghua University and Pengcheng Laboratory, trained across two phases on 1.4 trillion tokens total (438.8B in phase one on 24 GPUs, 960.0B in phase two on 96 GPUs), using blockwise FP8 with BF16/FP32 for sensitive operations.

For context, the same paper puts a rough number on what it would cost to reproduce competing open models with conventional infrastructure. "Training Llama3.2-3B costs over $1.5M in our estimation," the authors write, and reproducing SmolLM3-3B lands at "$719K." Both figures are the authors' own estimates, not third-party reproductions.

The recipe is more than budget engineering. The authors introduce MuonH, an extended Muon optimizer with "hyperball constraints on parameter weights and updates," and a curriculum model averaging scheme that "organizes training over coarse-grained data chunks according to configured source-local preferences and averages selected checkpoints." Blockwise FP8 yields a "1.34× quality-adjusted net gain," they report, and curriculum model averaging delivers "2.40× relative to uniform scaling curve."

A derived scaling law suggests "about $4.4K, less than $5,090, is sufficient to reach the performance of Qwen2-1.5B." That is the number the paper uses in its title.

The full recipe (data, code, and weights) is released under Apache 2.0.