Byteification retrofits Olmo, Qwen, Llama into byte LLMs

Converting an existing large language model to operate directly on bytes, rather than word fragments, took the authors just 49.1 billion tokens of further training, less than 1% of what a typical pretraining run burns. The paper, published in Nature on 7 October, introduces a procedure the authors call "byteification" and applies it to four open-weight checkpoints: Olmo 3 7B, OLMo 2 1B, Qwen3 8B Base, and Llama 3 8B. The byteified versions are named Bolmo 7B, Bolmo 1B, Bwen 8B, and Blama 8B.

The pitch is efficiency. Byte-level models have historically lagged subword-tokenized models despite their appeal for domains where single characters carry meaning. From the abstract: "Models that instead operate directly on the byte encoding of text avoid these limitations, but until now they have lagged behind subword-based models in performance. Here we introduce a general method for creating byte-level large language models through byteification that approach the capabilities of subword-based systems."

The standout number is Bolmo 7B's reported "+16.5% absolute improvement in STEM tasks" over BLT 7B, a prior byte-level approach. The paper also claims the method transfers instruction-following behaviour from the source model's post-training at zero cost via task arithmetic.

Authors on the paper include Benjamin Minixhofer, Luke Zettlemoyer, Noah A. Smith, Anna Korhonen, Edoardo M. Ponti, Luca Soldaini and Valentin Hofmann, with affiliations spanning the University of Edinburgh and the University of Washington. Two researchers in our tracker surfaced the Nature link.

The abstract reports that the byteified models "approach the capabilities of subword-based systems", which is softer wording than parity. No head-to-head accuracy tables against each source model appear in it, and no wall-clock inference numbers either.