Byteification retrofits Olmo and Qwen as byte-level LLMs

"Byteification," described in a paper published this month in Nature, converts large language models that run on subword tokens into byte-level models with roughly 49.1 billion tokens of additional training. The authors call that "less than 1% of a typical pretraining budget" compared to training a byte-level system from scratch.

Byteified Bolmo 7B, retrofitted from Olmo 3 7B, achieved "a +16.5% absolute improvement in STEM tasks over BLT 7B," one of the leading from-scratch byte-level baselines. Bwen 8B, derived from Qwen3 8B Base, outperformed all earlier publicly available byte-level models of comparable size.

The conversion runs in two stages, 9.8 billion tokens then 39.3 billion. Four retrofitted variants are described: Bolmo 7B, Bolmo 1B from OLMo 2 1B, Bwen 8B, and Blama 8B from Llama 3 8B. The 7B evaluation suite covers 40 benchmark tasks spanning code generation, mathematics, medical QA, and reading comprehension.

The paper's own claim is that byteified models reach performance "approaching their source subword-based systems" while beating them on character-level reasoning and holding practical inference speeds. The abstract does not publish latency numbers or per-task scores for the full 40-task suite. Two of the researchers we follow passed the link around the week it posted.