'Byteification' retrofits Llama 3, Qwen3 at under 1% cost

A new Nature paper from Benjamin Minixhofer and co-authors, including Luke Zettlemoyer and Noah A. Smith, describes 'byteification,' a two-stage retrofit that converts existing subword language models to operate directly on bytes using under 1% of typical pretraining cost: 49.1B tokens total across the full model suite. The team ships byte-level versions of Llama 3 8B, Qwen3 8B Base, OLMo 3 7B and OLMo 2 1B, named Blama 8B, Bwen 8B, Bolmo 7B and Bolmo 1B.

The byteified models reach "performance close to that of state-of-the-art subword-level LLMs," the paper reports, while substantially outperforming their subword counterparts on character-level reasoning. Bwen 8B shows a 16.5 percentage point absolute improvement on STEM tasks over BLT 7B, an earlier byte-level baseline. Evaluation runs across HumanEval, MBPP, DeepSeek LeetCode, GSM8K, Minerva MATH, MedMCQA, Natural Questions, SQuAD and the CUTE character-level suite.

Architecturally the authors introduce what they call a "latent tokenizer language model," pairing non-causal boundary prediction with a local encoder and decoder built from mLSTM layers for byte contextualization. The motivation is specific: byte-level models avoid the "information obscuration" of subword tokenization, which the paper argues matters most for computer code and biological sequences, where meaning depends on individual bytes.

Two researchers in the AI Weekly Who's Who directory shared the paper's link on publication. The authors describe the retrofit as reusing "the existing ecosystem around the source large language model" rather than rebuilding one from scratch.