Nature paper retrofits OLMo, Qwen3, Llama 3 to byte LLMs

Subword language models can be retrofitted to operate directly on raw bytes using 49.1 billion extra training tokens, what the authors call less than 1% of a typical pretraining budget, a new Nature paper from Benjamin Minixhofer and co-authors reports.

The retrofit runs in two stages: train a local encoder, decoder, and boundary predictor around the frozen original model, then fine-tune end to end. The team shipped four byteified variants: Bolmo 7B from OLMo 3, Bolmo 1B from OLMo 2, Bwen 8B from Qwen3 8B Base, and Blama 8B from Llama 3 8B.

On the headline comparison, Bolmo 7B posts a +16.5% absolute gain on STEM tasks over BLT 7B, the prior byte-level baseline trained from scratch. 'Byteification removes a long-standing performance barrier to end-to-end byte-level language modelling,' the paper argues, adding that the byteified models 'outperformed, on average, all earlier publicly available byte-level LLMs of comparable size.'

The practical draw is character-level reasoning on things subword tokenizers chunk awkwardly: spelling, source code, biological sequences. The paper frames the pitch as models that 'can scale competitively while offering advantages in domains requiring fine-grained textual understanding.' Two researchers we track had the paper moving through their feeds the day it posted.