Byteification retrofits subword LLMs to run on raw bytes

The paper, published in Nature on October 7, calls the method "byteification": take an existing subword-based large language model and retrofit it into one that reads raw bytes, using less than 1% of a typical pretraining budget.

The authors run the conversion in two stages. The first distills a byte-level model from its subword parent while the global transformer is frozen, over 9.8B tokens. The second fine-tunes end-to-end over 39.3B tokens. Total: 49.1B tokens.

They produce four byteified variants: Bolmo 7B from Olmo 3 7B, Bolmo 1B from OLMo 2 1B, Bwen 8B from Qwen3 8B, and Blama 8B from Llama 3 8B. The paper reports Bolmo 7B shows a "+16.5% absolute improvement in STEM tasks over BLT 7B", an earlier byte-level baseline. Parameter counts stay close to the originals, drifting between -0.7% (Bolmo 1B) and +4.5% (Bolmo 7B). Two researchers we track posted the Nature link in our Who's Who index.

The motivation is blunt. "Subword tokenization obscures fine-grained information, which is problematic, especially for scientific data," the authors write; models that "operate directly on the byte encoding of text avoid these limitations."

Architecturally, the byteified models wrap the global transformer in an mLSTM-based local encoder, a non-causal boundary predictor and a local decoder, with a 512-token output vocabulary combining bytes and boundary signals. The authors claim byteified models "can arbitrarily be further sped up by training with higher ratios of bytes per patch". The paper's publicly summarized results do not include inference latency figures.