Byteification retrofits subword LLMs to byte-level cheaply

Byteification, a two-stage procedure that converts existing subword-based language models into byte-level models, needs roughly 49.1 billion tokens of additional training, which the authors call less than 1% of typical pretraining budgets. The method is published in a 7 October Nature paper.

It yields three retrofitted models: Bolmo, Bwen and Blama. The paper puts Bolmo 7B at a 16.5 percentage-point absolute gain over BLT 7B on STEM tasks, the public byte-level reference the authors compare against. Bwen 8B, they write, "statistically significantly outperformed all earlier publicly available byte-level models of comparable size."

The authors, a group that includes Benjamin Minixhofer, Luke Zettlemoyer, Noah A. Smith, Luca Soldaini and Valentin Hofmann, argue that "models operating on raw text encodings can scale competitively while offering advantages in domains requiring fine-grained textual understanding," pointing to code and biological sequences as the cases that benefit. Two researchers we follow had already shared the link by the time this reached our queue.

Byteified models keep parameter counts close to their source checkpoints and, per the abstract, maintain "performance near source subword models" at what the authors call practical inference speeds. The abstract publishes no throughput numbers for the retrofitted variants, and the reported results sit at 7B and 8B scale.