Nature paper byteifies OLMo, Qwen3, Llama 3 at <1% budget

A method called "byteification," published in Nature on 7 October 2026, converts an existing subword language model into one that operates directly on bytes using 49.1 billion extra training tokens, which the authors characterise as less than 1% of a typical pretraining budget.

The authors, led by Benjamin Minixhofer and including Luke Zettlemoyer, Noah A. Smith, Luca Soldaini and Valentin Hofmann, shipped byteified variants of three open model families: Bolmo from OLMo, Bwen 8B from Qwen3, and Blama 8B from Llama 3. Bolmo 7B logged a +16.5% absolute improvement in STEM tasks over BLT 7B, the prior byte-level baseline.

Under the hood sits what the paper calls a "latent tokenizer language model" that "aggregates the byte stream into patches," then converts the output back to bytes. One technical distinction from earlier byte-level work: the retrofit uses "1 byte of future context" during prefill, where prior causal approaches did not.

The motivation is spelled out plainly. "Subword tokenization obscures fine-grained information," the paper argues, and the hit lands hardest on scientific data like computer code and biological sequences, where meaning depends on the individual characters or bytes.

Across the 7B evaluation suite the authors report results on 40 benchmarks. Two researchers we track posted the link within hours of publication.