Byteification turns Olmo, Qwen, Llama into byte-level LLMs

A method called byteification converts existing subword language models into byte-level ones at under 1% of a typical pretraining budget, according to a paper published in Nature on October 7 by Benjamin Minixhofer and co-authors. The retrofit retrains each source model on 49.1 billion extra tokens across two stages: a first stage that freezes the subword model and trains four new pieces for 9.8 billion tokens, and a second stage that unfreezes everything and runs end-to-end for another 39.3 billion.

From OLMo 3 7B, Qwen3 8B and Llama 3 8B the team produces Bolmo 7B, Bwen 8B and Blama 8B, with parameter counts shifting only 0.7% to 4.5% from the originals. Bolmo 7B shows a '+16.5% absolute improvement in STEM tasks over BLT 7B,' an earlier byte-level model trained from scratch.

Byte-level models read raw UTF-8 instead of word fragments, which the authors argue removes English-centric vocabulary constraints and improves character-level reasoning. In an Allen AI post, the team describes byteifying as a way to take 'an already capable subword model' and convert 'it into a byte-level one.'