Allen AI, Cambridge retrofit subword LLMs to byte-level

Bolmo 7B, a byte-level language model converted from OLMo, "achieved a +16.5% absolute improvement in STEM tasks over BLT 7B," according to a paper published 7 October 2026 in Nature by researchers at Allen AI and the University of Cambridge. BLT 7B is a prior byte-level baseline.

The team calls the technique "byteification" and describes it as "a two-stage conversion procedure to retrofit existing subword-based models into byte-level models with minimal extra training," consuming 49.1B tokens in total across the demonstration runs. They apply the method to three open-weight families: OLMo, Qwen3 8B, and Llama 3 8B, producing Bolmo 7B, Bolmo 1B, Bwen 8B, and Blama 8B.

Byte-level models operate on raw text encodings rather than subword tokens. Earlier byte-level approaches "lagged behind subword-based models in performance," the paper notes, and the authors frame their contribution as closing that gap. Byteifying "provides a key missing research direction by enabling the creation of state-of-the-art byte-level LLMs," they write, and report that the retrofitted models "substantially surpassed their subword-level counterpart regarding character understanding."

Two of the researchers we track had already shared the source link when the story reached our feed.