Ai2 retrofits OLMo, Llama, Qwen into byte-level models

A Nature paper from Benjamin Minixhofer and collaborators converts three families of open-weight language models (OLMo 3, Llama 3, and Qwen3) into byte-level counterparts using 49.1B training tokens total, which the authors describe as less than 1% of a typical pretraining budget. They call the procedure "byteification," and describe it as "a two-stage conversion procedure to retrofit existing subword-based models into byte-level models with minimal extra training."

The retrofits ship as Bolmo (from OLMo), Blama (from Llama 3 8B), and Bwen (from Qwen3 8B Base). Stage one trains local encoder and decoder components around a frozen global transformer; stage two unfreezes and runs end-to-end. On the 7B tier, Bolmo 7B posts a "+16.5% absolute improvement in STEM tasks" over the earlier byte-level BLT 7B across the paper's 40-benchmark suite. Of the three retrofits, Ai2's companion blog post calls Bwen 8B the strongest, "outperforming Bolmo 7B across our aggregate evaluation suite."

Byte-level language models have been pitched for years as the fix for subword tokenizers' blind spots: fragile handling of rare words, non-Latin scripts, code, and biological sequences where the characters inside tokens matter. The hurdle has been cost. Training a competitive byte-level model from scratch means paying the full pretraining bill, which is why the dominant open families all ship with subword tokenizers. The paper's claim is that most of that compute can be amortized from an existing checkpoint.

Two of the researchers in our Who's Who directory had the link circulating within days of publication. The authors describe their byteified models as able to "approach the capabilities of subword-based systems" while gaining ground on character-level understanding.