Allen Institute paper retrofits LLMs to byte-level

The paper, published in Nature, describes a method called "byteification" that converts a subword-based large language model into a byte-level model through a two-stage conversion procedure requiring "minimal extra training."

The retrofit used "49.1B tokens in total" across both stages. Bolmo 7B, built from Olmo 3 7B, delivered a "+16.5% absolute improvement in STEM tasks over BLT 7B," a previous byte-level model trained from random initialization. Bwen 8B, built from Qwen3 8B, "numerically improved in every category" except GenQA against earlier byte-level approaches.

Architecturally, bytes are aggregated into patches, processed by a transformer, then depooled back to bytes. The paper's key move is "non-causal patch boundary prediction" using one byte of future context during prefill, matching how subword tokenizers already behave even though prediction only uses past tokens. Stage one distills the source model's behavior with "a binary cross-entropy loss" on boundaries plus L2 distance matching. Stage two trains end to end on the full model.

The retrofit also inherits the parent model's post-training ecosystem. The authors report that task arithmetic merged instruction-following from post-trained Olmo 3 checkpoints into Bolmo "without any extra training cost," enabling zero-shot adaptation of the subword model ecosystem to byte-level variants. Earlier byte-level approaches "did not outperform the subword-level models" on general tasks; the paper claims these ones do.

Two researchers we track posted the paper the day it dropped.