'Byteification' converts subword LLMs to byte-level in Nature

A method published in Nature on October 7 converts existing subword-based language models into byte-level ones using what the authors describe as a "two-stage conversion procedure requiring minimal additional training (49.1B tokens)." The team, led by Benjamin Minixhofer, calls the technique "byteification."

Applied to three open bases, the procedure produces Bolmo 7B, Bwen 8B and Blama 8B. The paper reports that the byteified variants "approach the capabilities of subword-based systems" on standard tasks and "substantially surpassed their subword-level counterpart regarding character understanding."

Two design choices matter beyond the headline comparison. A non-causal boundary prediction mechanism closes the expressivity gap between byte streams and the subword vocabularies the source models were trained on. And post-trained subword weights, the authors write, can be ported into the byteified models "through task arithmetic without additional training," which offers a route to inherit instruction tuning and safety alignment without redoing them at the byte level.

The author list spans Luke Zettlemoyer, Noah A. Smith, Anna Korhonen, Edoardo M. Ponti and Luca Soldaini, several of the groups behind the OLMo, Qwen and Llama lineages the paper retrofits. Two names from our Who's Who tracker surfaced the paper on publication day.