Nature retrofit converts OLMo, Qwen3, Llama to byte-level LLMs

Byteification, a two-stage retrofit introduced in Nature, converts existing subword language models into byte-level ones using less than 1% of a typical pretraining budget. The authors, a group spanning the University of Cambridge, the University of Edinburgh and the Allen Institute for AI, apply the recipe to four open-weight bases and ship them: Bolmo 7B and Bolmo 1B from OLMo, Bwen 8B from Qwen3, and Blama 8B from Llama 3.

The headline figure, from the paper: Bolmo 7B, retrofitted from OLMo 3, posts a "+16.5% absolute improvement in STEM tasks over BLT 7B," the current byte-level baseline. The paper also reports that the retrofitted models "substantially surpassed their subword-level counterpart regarding character understanding," the pitch for byte-level operation in the first place.

The recipe itself is modest by frontier standards. Stage 1 trains a local encoder, decoder and boundary predictor around the frozen base for 9.8 billion tokens; Stage 2 fine-tunes the full stack end-to-end for a further 39.3 billion. Byteified models add between 0.7% and 4.5% parameters on top of the source. The data mix runs on roughly 172B tokens from Dolma 3 augmented with 75M CUTE-style tokens.

Two of the researchers we follow circulated the Nature link in the days after publication. The subword-versus-byte-level debate has been live in the field for years, and the claim on offer here is that labs no longer have to pick at pretraining time.