Nature paper retrofits subword LLMs to operate over bytes

A method published in Nature on 7 October 2026 converts existing subword-based language models into byte-level models using less than 1% of a typical pretraining budget, or 49.1B tokens in total. The authors call it "byteification" and demonstrate it on four open checkpoints, producing models they name Bolmo 7B (from OLMo 3 7B), Bolmo 1B (from OLMo 2 1B), Bwen 8B (from Qwen3 8B Base) and Blama 8B (from Llama 3 8B).

The pitch, in the paper's own words: "Subword tokenization obscures fine-grained information, which is problematic, especially for scientific data." The authors argue the problem goes further. "LLMs that use subword tokenization suffer from limited character-level understanding, which especially hinders performance with scientific data, such as code and biological sequences," they write in the Nature paper. They also note that fixed vocabularies "in practice leads to English-centricity."

Byteification is a two-stage procedure. Stage 1 trains local components only, on 9.8B tokens. Stage 2 does full end-to-end training, on 39.3B tokens. The headline comparison: Bolmo 7B achieved "a +16.5% absolute improvement in STEM tasks over BLT 7B," the paper reports. BLT is an earlier byte-level model trained from random initialization. The ancestry is the point. Prior byte-level efforts started cold; this one inherits a trained subword model's weights.

The paper also makes a scaling claim: "Byte-level LLMs do not suffer from the softmax bottleneck, enabling an unbounded increase in efficiency for a smooth drop-off in performance." Third-party replication of these specific retrofits does not appear in the published version. Two of the researchers we track had circulated the paper by the time we picked it up.