Nature paper retrofits subword LLMs to operate over raw bytes
TL;DR
- Byteification converts existing subword LLMs into byte-level models using 49.1 billion training tokens, under 1% of a typical pretraining run.
- Bolmo 7B, retrofitted from OLMo 3 7B, posted a 16.5% absolute improvement on STEM tasks over the prior byte-level BLT 7B baseline.
- The authors shipped byteified variants of three open families: Bolmo (from OLMo), Bwen 8B (from Qwen3) and Blama 8B (from Llama 3).
An existing subword language model can be retrofitted to read raw bytes directly for 49.1 billion extra training tokens, less than 1% of a typical pretraining budget, according to a Nature paper by Benjamin Minixhofer and co-authors at the Allen Institute for AI and the University of Cambridge.
The method, called byteification, runs in two stages. A first stage of 9.8B tokens freezes the global transformer and trains local encoder and decoder components, including a boundary predictor. A second stage of 39.3B tokens trains the model end-to-end.
The authors shipped byteified variants of three open families: Bolmo (from OLMo), Bwen 8B (from Qwen3) and Blama 8B (from Llama 3). On STEM tasks, Bolmo 7B posted a '16.5 percent absolute improvement' over BLT 7B, the prior byte-level baseline. According to coverage at bioengineer.org, the boundary predictor reached 'over 99 percent accuracy in emulating the original tokenizer'.
Two of the AI researchers we follow had circulated the Nature link within a day of publication.
Shared on Bluesky by 2 AI experts
-
Our paper on retrofitting language models to operate over bytes – the approach behind Bolmo – has been accepted to Nature! 🎉 We’re also releasing new checkpoints that extend our method from Olmo to Qwen & Llama. 🧵 buff…
View on Bluesky →
Originally reported by nature.com
Read the original article →Original headline: Retrofitting language models to operate over bytes - Nature