Nature paper retrofits Olmo and Qwen into byte-level LLMs

Nature has published a method the authors call 'byteification', a procedure for turning existing subword language models into byte-level ones for less than 1% of a typical pretraining budget.

The paper, led by Benjamin Minixhofer, applies the recipe to four open bases: Olmo 3 7B becomes 'Bolmo 7B', OLMo 2 1B becomes 'Bolmo 1B', Qwen3 8B Base becomes 'Bwen 8B', and Llama 3 8B becomes 'Blama 8B'. Training runs in two stages: 9.8B tokens (about 43B bytes) in the first, 39.3B tokens (about 173B bytes) in the second. 'Models that instead operate directly on the byte encoding of text avoid these limitations,' the authors argue, pointing at the character-level blind spots that subword tokenization introduces for code and biological sequences.

The headline result for Bolmo 7B is '+16.5% absolute improvement in STEM tasks over BLT 7B', the prior byte-level baseline. Across a 40-task 7B evaluation suite and a 22-task 1B suite covering HumanEval, MBPP, GSM8K, CUTE, EXECUTE, MMLU and ARC, the retrofitted models approach their subword parents while handling character-level reasoning more cleanly. The architecture uses a 'latent tokenizer language model (LTLM) with non-causal boundary prediction' that peeks 'up to 1 byte of future context'.

Two researchers we track shared the paper on publication day. The abstract names those benchmarks without publishing per-task breakdowns, and the recipe still depends on an already-trained subword model, so the saving is a conversion rather than a from-scratch byte-level training.