Byteification retrofits subword LLMs to raw-byte operation

A method called 'byteification' retrofits existing subword-based large language models into byte-level ones with less than 1% of the training typically needed to build such a model from scratch, which the paper quantifies at 49.1 billion tokens. Published this month in Nature, the paper is led by Benjamin Minixhofer with senior author Valentin Hofmann of LMU Munich, and releases four retrofitted checkpoints: Bolmo 7B and 1B from OLMo, Blama 8B from Llama 3 8B, and Bwen 8B from Qwen 3 8B.

Bolmo 7B shows a +16.5% absolute improvement on STEM tasks over BLT 7B, the strongest prior byte-level system. The abstract reports more broadly that the byteified models 'outperform earlier byte-level approaches and excel on character-level reasoning tasks,' reaching 'practical inference speeds' while reusing the ecosystem of each source model.

Hofmann puts the gain in character-level terms. 'Traditional LLMs struggle with tasks that require character-level capabilities, such as spelling a word backward. Our byteified models are substantively better at this,' he told TechXplore. He adds that 'a good representation of the low-level structure of text is critical in many areas of science, for example, when working with code or biological sequences.'

The problem byteification targets sits inside the tokenizer. Subword tokenization 'obscures fine-grained information, which is problematic, especially for scientific data,' the abstract reads, flagging computer code and biological sequences where 'meaning depends on the individual characters or bytes.' Models that work directly on bytes avoid this, but had 'lagged behind subword-based models in performance' until, in the authors' framing, this result.

The paper's claim is that its approach 'remove a long-standing performance barrier to end-to-end byte-level language modelling.' Beyond the Bolmo-versus-BLT STEM delta, the retrieved text does not surface per-benchmark tables, and no inference-cost comparison against the source subword checkpoints appears in the material covered here.