Bolmo byteifies subword LLMs, beats BLT by 16.5% on STEM

A method called "byteification" converts existing subword-based language models to operate directly on raw bytes, with the resulting Bolmo 7B posting a "+16.5% absolute improvement in STEM tasks over BLT 7B," according to a paper in Nature published October 7, 2026.

The retrofit requires "less than 1% of a typical pretraining budget (49.1B tokens)," split across a 9.8B-token stage one and a 39.3B-token stage two. During stage one the global transformer stays frozen while the authors plug in a local encoder of mLSTM layers, a non-causal boundary predictor with one byte of lookahead, and a local decoder, with the output vocabulary expanded from 256 to 512 bytes.

Across sizes, parameter overhead is modest: Bolmo 1B lands roughly 0.7% smaller than its subword source, while Bolmo 7B, Blama 8B, and Bwen 8B carry about 4.5%, 2.7%, and 1.5% more parameters respectively. On character-understanding benchmarks, the byteified models "substantially surpassed their subword-level counterpart," the paper reports, and across broader evaluations the method beats comparable byte-level efforts EvaByte 6.5B, TFree-HAT 7B, and BLT 7B. Two researchers we track flagged the paper the same day it ran.

The authors describe the goal as letting byte-level models "approach the capabilities of subword-based systems," with advantages for scientific domains like computer code and biological sequences. Training drew roughly 172B tokens from the Dolma 3 pretraining mix, supplemented with 75M tokens of character-understanding tasks to encourage fine-grained textual reasoning.