Byteification retrofits subword LLMs to byte-level inputs

A Nature paper introduces "byteification," a two-stage procedure that converts existing subword-tokenized language models into byte-level ones using less than 1% of a typical pretraining budget. The authors, from Edinburgh, Cambridge, the Allen Institute for AI and the University of Washington, applied it to three open checkpoints to produce Bolmo 7B, Blama 8B and Bwen 8B.

The pitch, in the Nature paper's own words, is that subword tokenization "obscures fine-grained information critical for scientific data like code or biological sequences." On character-understanding benchmarks the byteified models substantially outperform both their subword source checkpoints and prior byte-level architectures. Bolmo 7B scores a "+16.5% absolute improvement in STEM tasks over BLT 7B," and Bwen 8B shows "statistically significant outperformance over all earlier byte-level models."

The core move is a non-causal boundary predictor that peeks one byte ahead.

The authors frame this as a fix for an expressivity mismatch, noting that "subword tokenizers use information about future bytes to place token boundaries," while earlier byte-level language models had looked only backwards. Boundaries are parameterised by cosine distance between adjacent byte representations. Training runs in two stages: 9.8B tokens with the global transformer frozen for subword-to-byte distillation, then 39.3B tokens end-to-end, for 49.1B tokens total.

Two researchers we track in our directory flagged the paper on publication. The abstract reports practical inference speed comparable to subword models but publishes no per-model latency figures, and the near-parity claim is drawn against the authors' own source Olmo 3 checkpoints rather than independently reproduced.