Nature paper retrofits LLMs to byte-level at under 1% cost

A team led by researchers from the Allen Institute for AI, the University of Edinburgh and the University of Washington says it can convert existing subword-based large language models into byte-level ones using less than 1% of a typical pretraining budget, publishing the method in Nature on October 7. The paper reports that Bolmo 7B, byteified from Olmo 3 7B, delivers "a +16.5% absolute improvement in STEM tasks over BLT 7B," an earlier byte-level model.

The appeal of operating over bytes is granularity. Subword tokenization, the authors write, "obscures fine-grained information, which is problematic, especially for scientific data" such as computer code or biological sequences. Full byte-level training from scratch has historically been costly. The two-stage byteification process puts most of the work into a local encoder, decoder and boundary predictor while keeping the global model frozen, then runs end-to-end training totaling 49.1B tokens, roughly 43B plus 173B bytes across the two stages.

Four retrofits are presented: Bolmo 7B (from Olmo 3 7B), Bolmo 1B (from OLMo 2 1B), Bwen 8B (from Qwen3 8B Base) and Blama 8B (from Llama 3 8B). The parameter overhead ranges from a small decrease on Bolmo 1B, roughly 10M fewer at −0.7%, up to +4.5% for Bolmo 7B, roughly 330M more. Bwen 8B, the paper says, "outperformed all earlier publicly available byte-level models of comparable size."

The core efficiency framing is that byte-level models avoid "the softmax bottleneck" that limits subword-level LLMs, enabling "an unbounded increase in efficiency for a smooth drop-off in performance." The authors also show that a byteified base model can inherit instruction-following behavior from an already post-trained source through task arithmetic, with no extra training. The 7B evaluation suite covers 40 tasks including HumanEval, MMLU, GSM8K and the newly introduced CUTE and EXECUTE benchmarks for character-level understanding; the 1B suite covers 22.

Two of the researchers we follow in our Who's Who directory had the Nature link in their feeds on the day of publication.