Nature paper retrofits Llama, Qwen as byte-level LLMs

A Nature paper published October 7 introduces "byteification," a two-stage retrofit that converts existing subword language models into byte-level ones using "less than 1% of a typical pretraining budget (49.1B tokens in total)." The resulting Bolmo 7B, built from Olmo 3 7B, posts a "+16.5% absolute improvement in STEM tasks over BLT 7B," an earlier byte-level baseline.

The authors — Benjamin Minixhofer, Tyler Murray, Tomasz Limisiewicz, Anna Korhonen, Luke Zettlemoyer, Noah A. Smith, Edoardo M. Ponti, Luca Soldaini and Valentin Hofmann — frame the problem bluntly in the abstract: "Subword tokenization obscures fine-grained information, which is problematic, especially for scientific data—such as computer code or biological sequences—where meaning depends on the individual characters or bytes." Models that read raw bytes avoid that, but "until now they have lagged behind subword-based models in performance." The same recipe produces Bwen 8B from Qwen3 8B Base and Blama 8B from Llama 3 8B; the paper reports Bwen 8B "outperformed all comparable byte-level models statistically significantly." The paper is already circulating among two of the researchers we track in our Who's Who directory.

The training split is 9.8B tokens in stage one and 39.3B in stage two, drawn from Dolma 3 and augmented with 75M tokens of CUTE-style character data. Comparisons run against EvaByte 6.5B, TFree-HAT 7B and BLT 7B. The abstract publishes no per-detector inference-speed numbers, only the claim of "practical inference speeds by efficiently processing byte-level information."