Nature paper retrofits OLMo, Qwen3, Llama to byte-level LLMs

Converting an existing subword language model to run directly on UTF-8 bytes can be done on "less than 1% of typical pretraining budget (49.1B tokens total)," according to a paper published in Nature on 7 October 2026.

The team, with first author Benjamin Minixhofer, calls the procedure byteification and has already applied it to four open base models: Bolmo 7B from OLMo 3 7B, Bolmo 1B from OLMo 2 1B, Bwen 8B from Qwen3 8B Base, and Blama 8B from Llama 3 8B. Parameter counts land within 4.5% of the source model in each case.

The paper reports a "16.5% absolute improvement in STEM tasks over BLT 7B" for Bwen 8B against a byte-level baseline, and that "Byteified models substantially surpassed subword counterparts on character understanding." Two researchers we follow in our Who's Who directory posted the link.

Mechanically, the approach wraps a global transformer with local byte encoders and decoders, which the authors call a Latent Tokenizer Language Model, and uses non-causal boundary prediction with one byte of future context. The abstract does not break out per-benchmark results beyond the headline STEM figure, and gives no GPU-hour or dollar cost for the two-stage conversion.