Byte-level LLMs match subword rivals via Nature retrofit

Large language models can be retrofitted to operate directly on raw bytes instead of word-fragment tokens using fewer than 49.1 billion training tokens, which the authors call "less than 1% of typical pretraining budget", according to a method published in Nature on 7 October by Benjamin Minixhofer and co-authors. The team call the technique "byteification" and apply it to OLMo 3, OLMo 2, Qwen3 and Llama 3, yielding four new models they name Bolmo 7B, Bolmo 1B, Bwen 8B and Blama 8B.

The headline number is Bolmo 7B, which the paper reports gives "+16.5% absolute improvement in STEM tasks over BLT 7B", the strongest earlier open byte-level baseline. Bwen 8B, converted from Qwen3 8B, "outperformed all earlier publicly available byte-level LLMs of comparable size". Across a 40-task evaluation suite covering HumanEval, MMLU, GSM8K, Minerva MATH, DROP and MedMCQA, the byteified variants roughly match their subword counterparts, while substantially surpassing them on the character-level benchmarks CUTE and EXECUTE. A pair of researchers we track flagged the paper the day it went live.

The abstract is direct about what byte-level models have struggled with: "until now they have lagged behind subword-based models in performance". Byteification is a two-stage recipe. First, distillation on 9.8 billion tokens with the base model's global parameters frozen. Then end-to-end fine-tuning on 39.3 billion tokens. The authors also introduce a non-causal boundary predictor that uses "1 byte of future context" during prefill, so the byte model places token splits more like a subword tokenizer would.

Parameter overhead is small. Bolmo 7B is 4.5% larger than OLMo 3 7B, Bwen 8B 1.5% larger than Qwen3 8B. The authors claim "task arithmetic" transfers instruction-following from the subword checkpoints to the byteified models at "zero-cost" integration. The pitch is reuse rather than rebuild, pointed most clearly at domains where tokenizers actively obscure meaning: computer code, biological sequences, and multilingual text.