Byteification retrofits language models to operate on bytes

Byteification converts existing subword-based language models to operate directly on bytes, using 49.1B tokens, less than 1% of typical pretraining. That is the headline claim of a Nature paper by Benjamin Minixhofer, Tyler Murray, Tomasz Limisiewicz, Anna Korhonen, Luke Zettlemoyer, Noah A. Smith, Edoardo M. Ponti, Luca Soldaini, Valentin Hofmann and others.

The problem the paper names is specific. "Subword tokenization obscures fine-grained information, which is problematic, especially for scientific data—such as computer code or biological sequences—where meaning depends on the individual characters or bytes," the abstract reads. Byte-level models avoid that limitation, "but until now they have lagged behind subword-based models in performance."

Byteified Bolmo 7B posts a "+16.5% absolute improvement in STEM tasks over BLT 7B," according to the authors, and byteified models "substantially surpassed their subword-level counterpart regarding character understanding." The retrofit uses what the paper calls latent tokenizer language models, with non-causal boundary prediction that accesses one byte of future context, and sidesteps the softmax bottleneck that constrains subword vocabularies. The named checkpoints are Bolmo 7B and 1B, Bwen 8B, and Blama 8B.

The authors describe byteification as "a general method for creating byte-level large language models through byteification that approach the capabilities of subword-based systems." The hedged verb is theirs. Per-benchmark tables, inference-cost numbers, and head-to-head results against today's strongest subword baselines are not in the abstract. Two researchers on our Who's Who list flagged the paper the day it ran.