Nature paper retrofits subword LLMs to byte-level operation

Byte-level large language models can match the capabilities of state-of-the-art subword models after a retrofit costing less than 1% of a typical pretraining budget, according to new research in Nature. The authors call the technique "byteification", a two-stage conversion that uses 49.1 billion tokens of additional training to turn subword models into byte-level ones.

The retrofitted systems pull ahead of prior byte-level work. Bolmo 7B, converted from OLMo, delivered a "+16.5% absolute improvement in STEM tasks over BLT 7B," the paper reports. The 8B conversion of Qwen3, Bwen 8B, "statistically significantly outperformed all the earlier byte-level models and numerically improved in every category except GenQA."

The motivation is what tokenization loses. Subword tokenization "obscures fine-grained information, which is problematic, especially for scientific data—such as computer code or biological sequences—where meaning depends on the individual characters or bytes," the paper writes. Its summary line: "Byteification lets us train byte-level LLMs with a performance close to that of state-of-the-art subword-level LLMs at the 1B and 7B parameter scales."

The paper does not test scales beyond 8B parameters.