Nature paper retrofits subword LLMs to operate on bytes

A two-stage retrofit called "byteification" converts existing subword-based language models into byte-level variants using less than 1% of a typical pretraining budget, according to a paper published in Nature by Benjamin Minixhofer and co-authors.

The headline numbers: Bolmo 7B records a "+16.5% absolute improvement in STEM tasks over BLT 7B," and Bwen 8B "outperformed all the earlier byte-level models" in evaluation. The full procedure spent 49.1B training tokens, split into roughly 9.8B in stage one and 39.3B in stage two. The boundary predictor that stitches bytes back into subword-like units uses "up to 1 byte of future context" to resolve expressivity mismatches with the original tokenizer.

The motivation sits in one line of the abstract: "subword tokenization obscures fine-grained information, which is problematic, especially for scientific data."

The byteified models "substantially surpassed their subword-level counterpart regarding character understanding," the authors write, though the abstract itself hedges that the byte-level variants "approach the capabilities of subword-based systems" rather than match them outright. The author list includes Luke Zettlemoyer, Noah A. Smith, Anna Korhonen, Luca Soldaini and Valentin Hofmann, alongside first author Minixhofer.