Nature paper retrofits Olmo, Qwen, Llama to byte-level LLMs

Bolmo 7B, a byte-level model retrofitted from Olmo 3 7B, beats the earlier byte-level BLT 7B by 16.5 percentage points on STEM tasks, and gets there on less than 1% of a typical pretraining budget: 49.1 billion tokens in total, split across two stages. The result was published in Nature on October 7, 2026 by Benjamin Minixhofer and eight co-authors.

The method is called byteification. In the first stage, the authors freeze a trained subword model and train only four new pieces (a local encoder, a local decoder, a boundary predictor, and a language-modelling head) for 9.8 billion tokens. In the second stage, they unfreeze everything and train end-to-end for another 39.3 billion. The source models they byteified: Olmo 3 7B, OLMo 2 1B, Qwen3 8B Base, and Llama 3 8B, producing the Bolmo, Bwen, and Blama variants.

"We introduce a general method for creating byte-level large language models through byteification that approach the capabilities of subword-based systems," the paper states. On character-level understanding tasks, the byteified models "substantially surpassed their subword-level counterpart."

The architectural wrinkle is a non-causal patch boundary predictor. During prefill the model is allowed one byte of future context when choosing where to break the input into patches; during decoding, causality is enforced through a special boundary symbol. The authors describe this as restoring "the expressivity of subword-level LLM boundaries."

Two of the researchers we track in our Who's Who directory had already shared the paper's link.

The efficiency claim is softer than the quality claim. The method can be "arbitrarily further sped up by training with higher ratios of bytes per patch," which trades patch granularity for throughput. Per-token wall-clock inference numbers against the subword source models do not appear in the retrieved abstract.