Byteification retrofits subword LLMs to run on raw bytes

A method called byteification, published in Nature, converts existing subword-based language models into byte-level ones with 49.1B tokens of additional training, well below the trillions typically used for pretraining from scratch. Stage one used 9.8B tokens; stage two added 39.3B more.

The authors applied the recipe to produce two retrofits of existing open checkpoints, Bolmo 7B and Bwen 8B. Bolmo 7B "achieved a +16.5% absolute improvement in STEM tasks over BLT 7B," a baseline byte-level model trained from random initialization. Bwen 8B "statistically significantly outperformed all the earlier byte-level models" and "numerically improved in every category."

The motivation is practical. "Subword tokenization obscures fine-grained information, which is problematic," the authors write, flagging scientific data "where meaning depends on the individual characters or bytes." Byte-level models have long been attractive for exactly this reason and prohibitively costly to train from scratch.

The paper hedges on where the gains come from. The authors note that character understanding may be "acquired primarily through scale" and that the Olmo 3 source model was "probably trained on substantially more tokens than the other models" it was compared against. That qualifier limits how cleanly byteification itself can be credited for the headline numbers. The paper was already moving through the researcher circles on our radar by the time of posting, with two tracked analysts sharing it.