Byteification retrofits subword LLMs to read raw bytes

Byteification, a two-stage recipe for converting a trained subword language model into one that reads individual bytes, requires less than 1% of a typical pretraining run, roughly 49.1 billion tokens split 9.8B for stage one and 39.3B for stage two. The method appears in a paper published in Nature on October 7, 2026 by Benjamin Minixhofer and colleagues including Luca Soldaini, Noah A. Smith and Valentin Hofmann.

The team ran the recipe on four open base models, producing Bolmo 7B (from Olmo 3), Bolmo 1B (from OLMo 2), Bwen 8B (from Qwen3) and Blama 8B (from Llama 3). Bolmo 7B, the paper reports, shows a "+16.5% absolute improvement in STEM tasks over BLT 7B", the previous byte-level baseline.

The pitch for operating on bytes rather than subword tokens is scientific data. Subword tokenization, the authors write, "obscures fine-grained information problematic for scientific data like code and biological sequences," while raw-byte models until now have "lagged in performance."

Two of the researchers in our Who's Who directory shared the paper the day it posted. The abstract does not publish latency numbers against the subword parents, so what "practical inference speeds" translates to in production remains to be read inside the full paper.