Nature paper retrofits Llama, Qwen to operate on bytes

The authors retrofit four open-weight language models (Olmo 3 7B, OLMo 2 1B, Qwen3 8B Base and Llama 3 8B) so they operate on raw bytes instead of subword tokens.

Nature publishes the method under the name 'byteification', from a team that includes Benjamin Minixhofer, Luke Zettlemoyer, Noah A. Smith, Luca Soldaini and Valentin Hofmann. The converted models are called Bolmo 7B, Bolmo 1B, Bwen 8B and Blama 8B. The two-stage conversion training uses '49.1B tokens in total', which the paper puts at less than 1% of a typical pretraining budget. The architecture adds mLSTM layers for local encoding and decoding, and the authors use non-causal boundary prediction with '1 byte of future context' at prefill time.

The headline comparison is against BLT 7B, a prior byte-level system. Bolmo 7B posts a '+16.5% absolute improvement in STEM tasks' over BLT 7B. The paper also reports that 'character understanding improvements substantially exceeded subword counterparts', and that the byteified checkpoints approach the performance of the subword models they were converted from. Two researchers in our Who's Who directory had already shared the Nature link by the time this piece went up.

The practical pitch is efficiency and flexibility. Byte models, the authors argue, sidestep the softmax bottlenecks that constrain subword vocabulary expansion and open the door to an 'unbounded increase in efficiency' through variable bytes-per-patch ratios. Post-training by task arithmetic, the paper claims, adds no extra training cost.

The abstract does not publish per-benchmark numbers against the subword source models themselves, nor the inference-time cost at the compression ratios the authors advertise.