Byteification retrofits OLMo, Qwen, Llama to byte-level

A paper in Nature published on October 7 shows that an off-the-shelf subword language model can be converted to run on raw bytes with 49.1 billion extra training tokens, which the authors put at less than 1% of a typical pretraining budget. Benjamin Minixhofer and co-authors at Cambridge, Edinburgh and the Allen Institute for AI call the recipe byteification and ship four retrofitted variants: Bolmo 7B and Bolmo 1B from OLMo, Bwen 8B from Qwen3, and Blama 8B from Llama 3.

The method is a two-stage retrofit. First a subword-to-byte distillation runs for 9.8B tokens with the global model frozen; then 39.3B tokens of end-to-end training on a shared data mix. The architectural move is non-causal boundary prediction using one byte of future context during prefill, which matches subword-tokenizer expressivity while keeping decoding causal through explicit boundary symbols.

"Models operating on raw text encodings can scale competitively while offering advantages in domains requiring fine-grained textual understanding," the authors write. Bolmo 7B landed a +16.5% absolute improvement on STEM tasks over BLT 7B, the prior byte-level baseline, and Bwen 8B statistically significantly outperformed all earlier byte-level models in the comparison, which included EvaByte 6.5B and TFree-HAT 7B.

Parameter overhead is small: +4.5% for Bolmo 7B, +2.7% for Blama 8B, +1.5% for Bwen 8B.

A Nature news piece accompanying the paper flags the obvious party trick: a byteified model can count the letter 'i's in 'artificial intelligence', something most subword LLMs flunk because they only see tokens, not characters. The paper does not publish inference latency or serving-cost numbers, and the headline STEM gain is measured against another byte-level model rather than the strongest subword baselines.