Nature paper converts Llama, Qwen and OLMo to run on bytes

Benjamin Minixhofer and co-authors report in Nature that an existing subword language model can be converted to operate directly over raw bytes using fewer than 1% of the tokens a model of that size normally sees in pretraining. The authors call the technique byteification and apply it to OLMo 2 1B, OLMo 3 7B, Qwen3 8B Base and Llama 3 8B, producing byte-level variants they name Bolmo, Bwen and Blama. "Our results remove a long-standing performance barrier to end-to-end byte-level language modelling," the abstract states.

The headline figure is 49.1 billion tokens of extra training: 9.8 billion in a first stage that freezes the global transformer and warms up a boundary predictor, local encoder and local decoder, then 39.3 billion in an end-to-end second stage. Bolmo 7B, converted from OLMo 3 7B, lands 16.5 points above the earlier BLT 7B byte baseline on STEM tasks. Bwen 8B, derived from Qwen3 8B Base, is reported as statistically significantly outperforming all earlier publicly available byte-level models of comparable size. On character-understanding suites such as CUTE and EXECUTE the byteified models beat their subword starting points; on broader benchmarks they are described as approaching, rather than fully matching, the subword originals.

The retrofit framing matters as much as the numbers. Rather than training a byte model from scratch, the authors reuse the global transformer verbatim, and they show that reinforcement-learned post-training (OLMo 3 RL-Zero in their example) can be merged into a byteified variant via task arithmetic at zero additional training cost.