Nature method 'byteifies' subword LLMs for 1% of pretraining

A Nature paper published October 7 introduces 'byteification,' a two-stage retrofit that turns existing subword language models into byte-level models using 49.1B tokens of extra training, less than 1% of a typical pretraining budget. The authors release four converted models: Bolmo 7B from OLMo 3 7B, Bolmo 1B from OLMo 2 1B, Bwen 8B from Qwen3 8B Base, and Blama 8B from Llama 3 8B.

On STEM tasks, Bolmo 7B posts a +16.5% absolute improvement over the prior BLT 7B byte-level system, and the byteified models approached or matched their subword parents across standard benchmarks while substantially outperforming them on character-level work.

"Traditional LLMs struggle with tasks that require character-level capabilities, such as spelling a word backward. Our byteified models are substantively better at this," said Valentin Hofmann, a junior professor at LMU Munich and one of the paper's authors, in coverage at Tech Xplore.

The architecture stacks a local mLSTM encoder, a non-causal boundary predictor that reads one byte of future context, a pooling step into patches, the original global transformer, and a depooling byte-level decoder. Stage 1 trains over 9.8B tokens to recover the source model's behavior without full backpropagation through the global transformer; Stage 2 runs end-to-end training over the remaining 39.3B tokens. Two researchers in our tracker shared the paper the same week it dropped.

Hofmann frames the stakes beyond spelling tricks: "A good representation of the low-level structure of text is critical in many areas of science, for example, when working with code or biological sequences."