Bolmo 7B beats BLT 7B by 16.5% after byte-level retrofit

A new paper in Nature published October 7 introduces 'byteification,' a procedure that retrofits existing subword-based language models to operate directly on raw bytes, using 49.1B training tokens in total — what the authors describe as 'less than 1% of typical pretraining budget.' The source models they rebuild include OLMo 3 7B, OLMo 2 1B, Qwen3 8B and Llama 3 8B.

'Models that instead operate directly on the byte encoding of text avoid these limitations, but until now they have lagged behind subword-based models,' the authors write.

The headline number: Bolmo 7B, byteified from OLMo 3 7B, posts a '+16.5% absolute improvement in STEM tasks over BLT 7B.' Bwen 8B, derived from Qwen3 8B, outperformed all earlier publicly available byte-level models of comparable size across benchmark suites that include HumanEval, MMLU and GSM8K.

The architectural change is a boundary predictor that uses '1 byte of future context' during prefill, which the authors say resolves an expressivity mismatch that constrained earlier latent-tokenizer approaches. The two-stage recipe starts with distillation from a frozen source model at 9.8B tokens, then end-to-end fine-tuning with byte-level information at 39.3B tokens.

Benjamin Minixhofer is first author, with co-authors including Luke Zettlemoyer, Noah A. Smith, Anna Korhonen and Luca Soldaini. Two researchers we track posted the link within days of release.

The paper reports that byteified models 'approach the capabilities of subword-based systems' and offer an 'unbounded increase in efficiency' through adjustable bytes-per-patch ratios. The Nature abstract does not publish head-to-head accuracy deltas against the stronger subword source models for every benchmark.