Byteification retrofits Llama, Qwen, Olmo to byte-level

A method called "byteification" converts existing subword-based language models including Llama 3 8B, Qwen3 8B and Olmo 3 7B into byte-level models using less than 1% of a typical pretraining budget, according to a paper published in Nature by Benjamin Minixhofer, Valentin Hofmann, Luke Zettlemoyer, Noah A. Smith and co-authors on October 7, 2026.

The authors describe "a two-stage conversion procedure to retrofit existing subword-based models into byte-level models with minimal extra training." The full procedure consumed 49.1 billion tokens. Applied to Olmo 3 7B, the retrofitted model, which the paper calls Bolmo 7B, "achieved a +16.5% absolute improvement in STEM tasks over BLT 7B, which was trained from random initialization." The team also released Bolmo 1B (from OLMo 2 1B), Bwen 8B (from Qwen3 8B Base) and Blama 8B (from Llama 3 8B).

The byteified models "substantially surpassed their subword-level counterpart regarding character understanding," the paper reports — the kind of task (counting letters, spelling, awkward scripts) that tokenizer-based LLMs famously stumble on. The authors credit a "non-causal patch boundary prediction" step that peeks at up to one byte of future context when deciding where to split a sequence. The STEM comparison is drawn against BLT 7B trained from scratch rather than against the subword source model directly.