Nature paper 'byteifies' subword LLMs into byte-level models

A team led by Benjamin Minixhofer has published in Nature a method they call "byteification" that converts existing subword-based language models into byte-level models operating directly on UTF-8, without pretraining from scratch.

The authors apply the recipe to four open-weight bases, producing Bolmo 7B from Olmo 3, Bolmo 1B from OLMo 2, Bwen 8B from Qwen3 and Blama 8B from Llama 3. Training runs in two stages: a first stage of 9.8B tokens that freezes the global transformer and trains local encoder and decoder components, then a 39.3B-token end-to-end stage. Across the pipeline roughly 172B tokens are drawn from Dolma 3 plus 75M synthetic character-understanding tokens, and parameter counts shift between 0.7% and 4.5% from the source models.

On evaluation across more than 40 benchmarks including HumanEval, GSM8K, MMLU, CUTE and EXECUTE, the paper reports a "+16.5% absolute improvement in STEM tasks over BLT 7B" for Bolmo 7B, and says the converted models "approach the capabilities of subword-based systems" while beating prior byte-level variants. The architecture adds a non-causal boundary predictor using "1 byte of future context," which the authors argue sidesteps earlier byte-level designs that relied only on past information. Two researchers we track circulated the link soon after it ran.

Task arithmetic also carried instruction-following from post-trained subword models over to the byteified variants without additional training, suggesting the surrounding ecosystem of fine-tunes transfers across the conversion.