Allen AI retrofits OLMo, Qwen3, Llama 3 to byte-level models

A method the authors call "byteification" converts existing subword-based language models into byte-level ones for less than 1% of typical pretraining cost, researchers from the Allen Institute for AI, Cambridge, Washington, Edinburgh, Stuttgart and Potsdam report in a Nature paper published October 7. The team released byteified versions of three open families: Bolmo 7B and 1B from OLMo, Bwen 8B from Qwen3, and Blama 8B from Llama 3.

On STEM evaluations, Bolmo 7B posts a "+16.5% absolute improvement in STEM tasks over BLT 7B", a prior byte-level model of comparable size, with the full retrofit consuming "49.1B tokens in total". The authors frame the motivation plainly: "subword tokenization obscures fine-grained information, which is problematic, especially for scientific data". Computer code and biological sequences are the two examples they cite, both domains where individual characters carry meaning the tokenizer tends to smear.

Byte-level architectures have trailed subword ones at scale because training them from scratch is prohibitive. The paper's claim is that you can skip the from-scratch step. Two researchers we follow on our Who's Who list circulated the paper the day it published.

The authors close by arguing byteified models "scale competitively while offering advantages in domains requiring fine-grained textual understanding".