'Byteification' turns subword LLMs into byte-level ones

A paper in Nature from Benjamin Minixhofer and co-authors introduces 'byteification', a procedure that converts existing subword-based language models into byte-level ones using 49.1 billion additional training tokens.

The retrofitted models carry prefixed names. Olmo 3 7B becomes Bolmo 7B. Qwen3 8B becomes Bwen 8B. Llama 3 8B becomes Blama 8B. The paper reports a '+16.5% absolute improvement in STEM tasks' for Bolmo 7B over the earlier byte-level BLT 7B, and says Bwen 8B 'statistically significantly outperformed all comparable byte-level models.'

Why operate on bytes at all. Subword tokenization obscures fine-grained information critical for scientific data like code and biological sequences. 'Models that instead operate directly on the byte encoding of text avoid these limitations, but until now they have lagged behind subword-based models,' the authors write.

The procedure runs in two stages: 9.8 billion tokens of subword-to-byte distillation with the global model frozen, then 39.3 billion tokens of end-to-end training. A separate task-arithmetic step merges existing post-trained subword checkpoints into the byte-level versions at zero additional training cost.

The authors frame the work as removing a 'long-standing performance barrier to end-to-end byte-level language modelling.' Two researchers we track shared the paper on the day of publication.