Byteification retrofits LLMs to run on raw bytes, not tokens

Bolmo 7B, a byte-level language model retrofitted from Olmo 3 7B, scored a +16.5% absolute improvement on STEM tasks over BLT 7B, which was trained from random initialization. The method behind it, described in Nature, is called byteification: a two-stage retrofit that converts an existing subword-tokenized LLM into one that operates directly on bytes.

"Here we introduce a general method for creating byte-level large language models through byteification that approach the capabilities of subword-based systems," the paper states. The authors frame the result as removing "a long-standing performance barrier to end-to-end byte-level language modelling."

The procedure runs in two stages. In the first, the authors train the local encoder, local decoder, boundary predictor and language-modelling head against an objective designed, in their words, around "exactly recovering the behaviour of the source subword-level LLM." The second stage trains the full model to use byte-level information directly. Total pretraining investment for byteification is 49.1B tokens; the full data mix across both stages is approximately 172B tokens drawn from Dolma 3, augmented with 75M character-understanding tokens.

Four byteified models accompany the paper: Bolmo 1B from OLMo 2 1B, Bolmo 7B from Olmo 3 7B, Bwen 8B from Qwen3 8B Base, and Blama 8B from Llama 3 8B. The abstract reports they "excel on character-level reasoning tasks" while "achieving practical inference speeds." No per-benchmark numbers appear beyond the Bolmo 7B STEM figure, and the abstract does not quantify what "practical" means against the subword baselines. Two researchers we track shared the paper's link this week.