Byteification retrofits OLMo, Qwen3, Llama 3 to run on bytes

A new Nature paper claims an existing subword language model can be converted to operate directly on bytes with 49.1 billion extra training tokens, less than 1% of a typical pretraining budget.

The method, called 'byteification', is described in a paper published in Nature on 7 October 2026 by Benjamin Minixhofer, Tyler Murray, Luke Zettlemoyer, Noah A. Smith, Luca Soldaini, Valentin Hofmann and co-authors. The team applied the recipe to three open model families and shipped four variants: Bolmo 7B and Bolmo 1B from OLMo, Bwen 8B from Qwen3, and Blama 8B from Llama 3.

The pitch rests on a long-standing trade-off. 'Models that instead operate directly on the byte encoding of text avoid these limitations, but until now they have lagged behind subword-based models in performance,' the authors write.

Byteification closes much of that gap. The paper reports Bolmo 7B achieved a '+16.5% absolute improvement in STEM tasks over BLT 7B,' the prior byte-level baseline. The design hinges on non-causal boundary prediction using one byte of future context during prefill, which the authors say matches subword-tokenizer expressivity while keeping causal decoding intact through explicit boundary symbols.

The abstract foregrounds the STEM delta against BLT but does not publish head-to-head numbers against each original source model across the broader benchmark grid, and gives no raw inference throughput figures. A couple of researchers we follow had the paper in rotation on release day.