Nature paper retrofits subword LLMs into byte-level models

Researchers have retrofitted off-the-shelf subword language models into byte-level models for less than 1% of a typical pretraining budget, or 49.1B tokens in total, according to a paper in Nature.

The method, which the authors call "byteification," is a two-stage conversion of existing subword-based LLMs. It produced Bolmo 7B, Bolmo 1B, Bwen 8B and Blama 8B. The paper reports that Bolmo 7B "achieved +16.5% absolute improvement in STEM tasks over BLT 7B," and that Bwen 8B "outperformed all the earlier byte-level models and numerically improved in every category except GenQA."

The abstract puts the motivation bluntly: "Subword tokenization obscures fine-grained information, which is problematic, especially for scientific data—such as computer code or biological sequences—where meaning depends on the individual characters or bytes." Byte models avoid that, but "until now they have lagged behind subword-based models in performance."

"Byteification establishes a connection between existing subword-level LLMs and byte-level LLMs," the authors write, framing the result as removing "a long-standing performance barrier to end-to-end byte-level language modelling."

A quieter claim is buried in the method: post-trained subword checkpoints can be merged into byteified models using task arithmetic "with zero extra training cost," meaning instruction tuning done on the source model carries over without retraining. Two researchers we track circulated the paper the same day.

The abstract does not publish per-benchmark inference latency numbers against the subword source models, and the strongest performance figures are reported against earlier byte-level work rather than against frontier closed-source subword systems.