Nature paper retrofits OLMo, Llama, Qwen to run on bytes

Researchers have converted four open language models to byte-level operation using less than 1% of a typical pretraining budget, in a paper published in Nature on 7 October 2026. The method, which Benjamin Minixhofer and colleagues call "byteification," applies a two-stage conversion procedure to produce Bolmo 7B (from OLMo 3 7B), Bolmo 1B (from OLMo 2 1B), Bwen 8B (from Qwen3 8B Base) and Blama 8B (from Llama 3 8B).

The team reports that Bolmo 7B achieved a "+16.5% absolute improvement in STEM tasks over BLT 7B," an earlier byte-level baseline, and that byteified models "substantially surpassed their subword-level counterpart regarding character understanding." Total training budget comes to roughly 49.1 billion tokens: 9.8 billion with the global model frozen in stage one, then 39.3 billion end-to-end. The architecture gives the model access to "1 byte of future context" during prefill, which the authors describe as addressing "a mismatch between the expressivity of subword tokenization and the latent LTLM tokenization."

Instruction-following can be transferred from the original subword checkpoints without retraining, via task arithmetic. The reported STEM gain is measured against BLT 7B rather than against the strongest subword models, and the abstract carries no per-task inference-speed numbers.