Nature paper retrofits Olmo, Llama, Qwen into byte-level LLMs

A new Nature paper reports that a trained subword-based language model can be converted into a byte-level one using less than 1% of a typical pretraining budget, or 49.1 billion tokens.

The method, which the authors call byteification, is described in Retrofitting language models to operate over bytes, published in Nature on October 7, 2026 by Benjamin Minixhofer and co-authors at the University of Cambridge, the University of Edinburgh, and the Allen Institute for AI. The procedure is two-stage: train a local encoder, decoder and boundary predictor around the frozen original model, then fine-tune end to end.

The paper applies this to three of the better-known open model families. Olmo 3 7B and OLMo 2 1B become Bolmo 7B and Bolmo 1B; Qwen3 8B Base becomes Bwen 8B; Llama 3 8B becomes Blama 8B.

On results, the paper reports "Bolmo 7B achieving a +16.5% absolute improvement in STEM tasks over BLT 7B," the prior byte-level baseline that was trained from random initialization. The retrofitted models "outperformed, on average, all earlier publicly available byte-level LLMs of comparable size." The authors also report that "Bwen 8B outperformed Bolmo 7B, exhibiting comparable trends and reaching performance close to and sometimes surpassing the source Qwen model."

Byte-level LLMs have historically trailed subword-based ones. The paper's claim is that the gap can be closed cheaply by starting from a strong subword model rather than training a byte-level system from scratch. Two of the researchers we track posted the Nature link the day it went live.

The paper's headline comparisons are STEM tasks against the prior byte-level baseline and character understanding against the subword source — a direct general-reasoning comparison between, say, Bwen 8B and Qwen3 8B Base across standard benchmarks isn't the frame the authors chose.