Byteification turns Olmo, Llama, Qwen into byte-level LLMs

A new Nature paper from Benjamin Minixhofer and colleagues at the Allen Institute for AI and the University of Cambridge introduces a method called byteification that converts existing subword language models to operate directly over bytes, using fewer than 1% of the tokens a byte-level model would normally need to pretrain. The flagship conversion, Bolmo 7B, reports a 16.5-point absolute improvement on STEM tasks over the BLT 7B byte-level baseline while recovering the behaviour of the source Olmo 3 7B.

Byteification is a two-stage recipe: train a local encoder, decoder and boundary predictor around the frozen original model, then fine-tune end to end. The authors apply it to four open-weight bases — Olmo 3 7B, OLMo 2 1B, Qwen3 8B Base and Llama 3 8B — producing Bolmo, Bwen and Blama variants on a conversion budget of 49.1 billion tokens. The boundary predictor emulates the original tokenizer with over 99% accuracy.

"Traditional LLMs struggle with tasks that require character-level capabilities, such as spelling a word backward," co-author Valentin Hofmann of LMU Munich said in a Tech Xplore writeup. "Our byteified models are substantively better at this." The paper argues that the approach "has the potential to overcome some long-standing shortcomings of LLMs," pointing specifically to code and biological sequences, where, as Hofmann puts it, "a good representation of the low-level structure of text is critical."

The hitch has long been cost. Byte-level pretraining from scratch is prohibitive at frontier scale, which is why most production LLMs still ship a subword tokenizer that fuses letters into opaque chunks — ChatGPT's tokenizer, by Hofmann's example, splits "LMU München" into "LM," "U," and "München." Starting from an already-trained subword model and bolting on byte-level machinery sidesteps that bill. The link was circulating among a couple of the researchers we track on the day it landed.