Nature paper retrofits Olmo, Qwen, Llama into byte-level LLMs

A paper in Nature on October 7 describes a method the authors call "byteification," which turns an existing subword language model into a byte-level one without pretraining from scratch. Benjamin Minixhofer and colleagues apply it to Olmo 3, Llama 3 and Qwen3, producing byte-level counterparts named Bolmo, Blama and Bwen.

The recipe is a two-stage procedure: "train a local encoder, decoder and boundary predictor around the frozen original model, then fine-tune end to end." The authors report that the full retrofit uses "less than 1% of a typical pretraining budget." Training ran on roughly 172B tokens from Dolma 3, with 9.8B tokens in the first stage and 39.3B in the second, plus 75M tokens of CUTE-style data.

The headline number: Bolmo 7B records a "+16.5% absolute improvement in STEM tasks over BLT 7B," the current byte-level baseline.

The motivation, as the authors put it, is that "subword tokenization obscures fine-grained information, which is problematic, especially for scientific data," including code and biological sequences. Their pitch: "Models operating on raw text encodings can scale competitively while offering advantages in domains requiring fine-grained textual understanding." The abstract does not publish per-token inference costs for the retrofitted models.

Two researchers from our expert tracker had shared the paper by midweek.