Nature paper 'byteifies' open LLMs to run on raw UTF-8 bytes

A new paper in Nature describes a procedure the authors call 'byteification': a way to turn an existing subword-based language model into one that reads and writes raw UTF-8 bytes, using under 1% of a typical pretraining budget. The team's retrofitted models are named Bolmo 7B, Bwen 8B and Blama 8B.

The motivation is stated plainly. 'Subword tokenization obscures fine-grained information, which is problematic, especially for scientific data—such as computer code or biological sequences—where meaning depends on the individual characters or bytes,' the authors write. Bytes preserve what a tokenizer averages away.

The training recipe runs in two stages totaling 49.1 billion tokens. In stage one, roughly 9.8B tokens, the local encoder and decoder are initialized randomly while the global transformer stays frozen, and the model learns to imitate its subword parent through a distillation loss. In stage two, roughly 39.3B tokens, the full model trains end to end on bytes.

The headline comparison pits Bwen 8B against BLT 7B, a byte-level baseline trained from scratch: Bwen posts a +16.5% absolute improvement on STEM tasks. Character-level evaluation runs on the CUTE and EXECUTE benchmarks, where byteified models substantially beat their subword sources. On the 40-task suite used for the 7B models, the retrofits land at near-source-model performance.

'Our results remove a long-standing performance barrier to end-to-end byte-level language modelling, demonstrating that models operating on raw text encodings can scale competitively,' the authors conclude. Two of the researchers we follow have already shared the paper's link. The retrieved summary gives one headline STEM figure and no per-task table for the general-benchmark parity claim.