Nature paper retrofits Llama, Qwen, OLMo into byte-level LLMs

A new Nature paper claims to convert existing subword language models into byte-level models for less than 1% of typical pretraining cost. The method, called 'byteification,' is applied to OLMo, Qwen3 and Llama 3 to produce Bolmo 7B and 1B, Bwen 8B and Blama 8B, with 49.1 billion training tokens in total.

The authors describe "a two-stage conversion procedure to retrofit existing subword-based models into byte-level models with minimal extra training." Stage one freezes the base model and trains only four new pieces for 9.8 billion tokens; stage two unfreezes everything and runs end-to-end for another 39.3 billion. The author list is Benjamin Minixhofer, Tyler Murray, Tomasz Limisiewicz, Anna Korhonen, Luke Zettlemoyer, Noah A. Smith, Edoardo M. Ponti, Luca Soldaini and Valentin Hofmann.

On the benchmarks the paper reports, Bolmo 7B hits "a +16.5% absolute improvement in STEM tasks over BLT 7B," a byte-level model trained from random initialization. Bwen 8B is described as reaching performance close to and sometimes surpassing the source Qwen model.

Two of the researchers tracked in our Who's Who index were already circulating the link.

The retrieved abstract does not quote per-task numbers for Blama 8B, inference-latency comparisons against the source subword checkpoints, or multilingual and code-benchmark results.