Byteification retrofits Olmo, Qwen, Llama to byte-level LLMs

A team including researchers at the Allen Institute for AI, Cambridge and Edinburgh say they have retrofitted several open-weight language models to operate directly on bytes rather than subword tokens, closing a gap that has long made byte-level systems impractical. Writing in Nature, the authors describe a two-stage procedure they call 'byteification' that converts models including Olmo 3, Qwen3 and Llama 3 into byte-level counterparts using roughly 49.1 billion tokens of extra training, which the paper puts at 'less than 1% of typical pretraining budgets.'

The authors argue this matters most where tokenization throws away structure. 'Subword tokenization obscures fine-grained information, which is problematic, especially for scientific data,' the abstract states, flagging code and biological sequences as cases 'where meaning depends on the individual characters or bytes.' Their 7B retrofit, Bolmo 7B, posts a '+16.5% absolute improvement in STEM tasks over BLT 7B,' an earlier byte-level architecture.

The core mechanism pairs a boundary predictor using '1 byte of future context' during prefill with a subword-to-byte distillation stage that freezes global parameters before end-to-end training. A separate benefit the authors flag is ecosystem reuse: byteified models can inherit instruction-tuned behavior from subword parents through 'task arithmetic merging with existing post-trained checkpoints,' avoiding another round of instruction tuning.

The paper reports headline wins on character-understanding tasks like CUTE and EXECUTE and on STEM evaluations, with baselines consisting of each model's own subword original plus the earlier BLT architecture rather than frontier closed systems. The byteified lineup also includes Bolmo 1B from OLMo 2, Bwen 8B from Qwen3, and Blama 8B from Llama 3.