Nature method retrofits Llama 3, Qwen3 as byte-level LLMs

A new paper in Nature claims it can convert existing subword language models into byte-level ones for less than 1% of a typical pretraining budget. The authors call the method 'byteification.'

It was applied to four open checkpoints: Olmo 3 7B became Bolmo 7B, OLMo 2 1B became Bolmo 1B, Qwen3 8B Base became Bwen 8B, and Llama 3 8B became Blama 8B. Total extra training: '49.1B tokens in total.'

The headline number is on STEM. 'Bolmo 7B achieved a +16.5% absolute improvement in STEM tasks over BLT 7B, which was trained from random initialization,' the paper reports. Models were evaluated on HumanEval, MBPP, DS-1000, MultiPL-E, GSM8K, MMLU, DROP, SQuAD, and CUTE, a character-level understanding benchmark.

The authors' framing of why bytes matter is deadpan. 'LLMs that use subword tokenization suffer from limited character-level understanding, which especially hinders performance with scientific data, such as code and biological sequences,' they write. They also argue byte-level LLMs 'do not suffer from the softmax bottleneck,' giving them an efficiency ceiling that fixed subword vocabularies lack.

The paper flags a caveat on method generality: 'A subtle requirement when post-training byteified models through task arithmetic is that the embedding values of the post-trained models need to be resettable to the values of the base model embeddings without a large performance loss, which is generally, but not always, the case.' Two researchers from our Who's Who tracker posted the paper's link on the day it ran.