Nature paper retrofits Olmo, Qwen, Llama to read raw bytes

A Nature paper published October 7 presents "byteification," a two-stage retraining process that converts existing subword language models into byte-level ones, and demonstrates the method on four open-source bases: Olmo 3 7B, OLMo 2 1B, Qwen3 8B Base and Llama 3 8B. The authors report that the resulting Bolmo 7B stays "close to matching" its source model on standard benchmarks while beating the prior byte-level system BLT 7B by "+16.5% absolute improvement in STEM tasks."

The motivation, in the authors' words, is that "subword tokenization obscures fine-grained information, which is problematic, especially for scientific data." Their architecture pairs a latent tokenizer language model with local encoder and decoder modules built from mLSTM layers and non-causal boundary prediction using one byte of lookahead. Training runs in two stages: 9.8 billion tokens of source-model emulation, then 39.3 billion tokens of end-to-end learning, on top of roughly 172 billion tokens from the Dolma 3 pretraining mix augmented with 75 million tokens of character-understanding tasks.

The team evaluates the 7B models across 40 benchmarks including HumanEval, MMLU, GSM8K, DROP and the character-level suites CUTE and EXECUTE. They also show task arithmetic can transfer instruction-following from post-trained Olmo 3 checkpoints to Bolmo without additional end-to-end tuning.

The broader pitch is that byte-level models offer "potential advantages for computational efficiency by reducing energy and deployment costs, for reducing biases introduced by English-centric subword tokenization and for enabling applications that require fine-grained textual understanding." The retrofit angle is the practical twist: the economics change once a byte-level model can be obtained by continuing from an existing checkpoint rather than pretraining from scratch.