Byteification turns Olmo, Qwen, Llama into byte-level LLMs

Converting an existing subword language model into a byte-level one takes less than 1% of a typical pretraining budget, according to a paper published in Nature on 7 October by Benjamin Minixhofer and co-authors.

The method, which the authors call "byteification," trains a local encoder, decoder and boundary predictor around the frozen original model, then fine-tunes end to end. Across the two stages the paper reports 49.1B tokens of training in total. Applied to open Olmo checkpoints, it produced Bolmo 1B and Bolmo 7B; the same recipe turned Qwen3 8B into Bwen 8B and Llama 3 8B into Blama 8B. The retrofit adds between 1.5% and 4.5% parameters.

On STEM tasks Bolmo 7B posts a "+16.5% absolute improvement" over BLT 7B, the byte-level baseline the paper uses for comparison. The authors motivate the work by what subword tokenization costs scientific users: it "obscures fine-grained information, which is problematic, especially for scientific data—such as computer code or biological sequences."

Two researchers we track in our Who's Who directory shared the Nature link on publication day.