Nature paper retrofits Olmo and Qwen into byte-level LLMs

A paper published in Nature describes a way to convert existing subword language models into byte-level ones for less than 1% of a normal pretraining budget: 49.1 billion tokens in total.

The method is called byteification. "We created a process we call byteifying, which takes an already capable subword model and converts it into a byte-level one with a relatively short additional training run," the team writes in a companion post. The abstract describes it as "a two-stage conversion procedure to retrofit existing subword-based models into byte-level models with minimal extra training."

Stage one spends 9.8 billion tokens, roughly 43 billion bytes, training a local encoder, decoder, boundary predictor and language-modeling head to mimic the frozen subword model. Stage two spends 39.3 billion tokens, roughly 173 billion bytes, fine-tuning the whole stack end-to-end so it exploits byte-level information directly.

Applied to four open bases, the recipe yields Bolmo 1B and Bolmo 7B (from OLMo 2 and Olmo 3), Bwen 8B (from Qwen3) and Blama 8B (from Llama 3). The authors report Bolmo 7B at a "+16.5% absolute improvement in STEM tasks over BLT 7B," the current byte-level baseline, and Bwen 8B that "statistically significantly outperformed all the earlier byte-level models." The team calls these "the first fully open byte-level language models competitive with strong subword models across a broad range of tasks."

The pitch for operating on bytes is prosaic. Tokenizers obscure the letters inside words, code, and biological sequences; byte-level models read the raw bytes that computers already store. The paper argues the approach carries "substantial promise as a foundation for future language models, with potential advantages for computational efficiency," particularly in "scientific and technical domains."

The retrieved abstract does not publish per-task inference latency, and the headline STEM number is measured against BLT 7B rather than against a strong subword rival at the same scale.