Nature paper byteifies OLMo, Qwen, Llama into byte-level LLMs
TL;DR
- A Nature paper reports the full byteification retrofit cost 'less than 1% of a typical pretraining budget (49.1B tokens in total).'
- Bwen 8B, converted from Qwen3 8B Base, posted '+16.5% absolute improvement in STEM tasks over BLT 7B.'
- The team released four byteified variants: Bolmo 7B, Bolmo 1B, Bwen 8B and Blama 8B, built on OLMo, Qwen3 and Llama 3.
A team led by Benjamin Minixhofer has published in Nature a method they call 'byteification' that converts existing subword language models into byte-level versions for 'less than 1% of a typical pretraining budget (49.1B tokens in total).'
The output is four retrofitted models: Bolmo 7B from OLMo 3 7B, Bolmo 1B from OLMo 2 1B, Bwen 8B from Qwen3 8B Base, and Blama 8B from Llama 3 8B. The headline figure is Bwen 8B's '+16.5% absolute improvement in STEM tasks over BLT 7B.'
The design wraps the frozen base model in a local encoder and decoder built from mLSTM layers, plus a non-causal boundary predictor given 'up to 1 byte of future context.' The authors justify the look-ahead by observing that 'subword tokenizers use information about future bytes to place token boundaries.' Training runs in two stages, 9.8B tokens then 39.3B tokens.
Two of the researchers in our directory flagged the paper. The Allen Institute's companion write-up describes the trick plainly as 'byteifying, which takes an already capable subword model and converts it into a byte-level one.'
Shared on Bluesky by 2 AI experts
-
Our paper on retrofitting language models to operate over bytes – the approach behind Bolmo – has been accepted to Nature! 🎉 We’re also releasing new checkpoints that extend our method from Olmo to Qwen & Llama. 🧵 buff…
View on Bluesky →
Originally reported by nature.com
Read the original article →Original headline: Retrofitting language models to operate over bytes - Nature