Nature paper closes byte-level vs subword LLM performance gap
TL;DR
- A new method called byteification retrofits existing subword language models into byte-level ones using less than 1% of a typical pretraining budget.
- Bolmo 7B posts a +16.5% absolute improvement on STEM tasks over the earlier byte-level model BLT 7B, the paper reports.
- The two-stage conversion was tested on Bwen 8B, Blama 8B, EvaByte 6.5B and TFree-HAT 7B, with 49.1B tokens of training in total.
Retrofitting an existing subword language model into a byte-level one takes less than 1% of a typical pretraining budget and closes the long-standing performance gap between the two approaches, researchers led by Benjamin Minixhofer write in Nature. The paper calls the two-stage conversion 'byteification' and reports 49.1B tokens of training in total.
The paper's Bolmo 7B shows a "+16.5% absolute improvement in STEM tasks over BLT 7B," an earlier byte-level baseline. Byte-level operation matters most, the authors argue, "especially for scientific data—such as computer code or biological sequences—where meaning depends on the individual characters or bytes." Other models tested in the paper include Bwen 8B, Blama 8B, EvaByte 6.5B and TFree-HAT 7B.
The authors frame the result as removing "a long-standing performance barrier to end-to-end byte-level language modelling." The abstract reports the STEM delta against BLT 7B but no multilingual, long-context or per-task inference numbers.
Shared on Bluesky by 2 AI experts
-
Our paper on retrofitting language models to operate over bytes – the approach behind Bolmo – has been accepted to Nature! 🎉 We’re also releasing new checkpoints that extend our method from Olmo to Qwen & Llama. 🧵 buff…
View on Bluesky →
Originally reported by nature.com
Read the original article →Original headline: Retrofitting language models to operate over bytes - Nature