Nature paper retrofits Llama, Qwen, OLMo to byte-level LLMs

"Byteification" turns subword-based language models like OLMo, Llama and Qwen into byte-level versions using 49.1B training tokens, what the authors call "less than 1% of a typical pretraining budget." The method, published in Nature on October 7 by a team spanning the University of Edinburgh and the University of Washington, uses "a two-stage conversion procedure...with minimal extra training."

The headline comparison: Bolmo 7B, the byteified OLMo variant, "achieved a +16.5% absolute improvement in STEM tasks over BLT 7B," an earlier byte-level architecture. Across the family of retrofitted models (Bolmo from OLMo, Blama from Llama, Bwen from Qwen), parameter counts shift between 0.7% smaller and 4.5% larger versus the subword originals.

Technically, the paper doubles the byte vocabulary from 256 to 512, folding in boundary-marked variants so that the boundary overhead is "effectively zero-cost." Stage 1 of training runs on 9.8B tokens, Stage 2 on 39.3B, with "75M tokens of CUTE-style data" mixed in for character understanding. The authors also demonstrate that existing post-trained subword checkpoints can be merged into byteified weights via task arithmetic, inheriting instruction-following behavior without additional training.