Nature method retrofits Llama, Qwen and OLMo to byte-level

A method called "byteification" converts existing subword language models into byte-level versions for less than 1% of typical pretraining cost, according to a paper published in Nature on October 7. The retrofitted Bolmo 7B, built from OLMo 3 7B, posts a +16.5% absolute improvement on STEM tasks over comparable byte-level models, while approaching the source subword model's broader benchmark scores.

Authors Benjamin Minixhofer, Luke Zettlemoyer, Noah A. Smith and colleagues apply the recipe to four model bases: OLMo 3 7B becomes Bolmo 7B, OLMo 2 1B becomes Bolmo 1B, Qwen3 8B becomes Bwen 8B, and Llama 3 8B becomes Blama 8B. The conversion uses non-causal boundary prediction with one byte of future context, mLSTM layers for local encoding and decoding, and a vocabulary expanded from 256 to 512 bytes through boundary fusion.

Training runs in two stages. Stage one freezes the global model and trains for 9.8B tokens (43B bytes); stage two then runs end-to-end for 39.3B tokens (173B bytes). The authors motivate the shift bluntly, writing that "subword tokenization obscures fine-grained information," the kind of granularity that matters for scientific text, code, and character-level manipulation tasks evaluated on CUTE and EXECUTE.

The 7B evaluation covers 40 tasks including HumanEval, MBPP, GSM8K and MMLU, with the 1B suite covering 22 core tasks. The paper frames the architecture work as resolving a "mismatch between the expressivity of subword tokenization and latent LTLM tokenization," and claims the design can "arbitrarily be further sped up by training with higher ratios of bytes per patch." Inference-latency numbers against the source subword models were not surfaced in the material retrieved here.