Bolmo 7B retrofits subword LLMs onto byte-level in Nature

A paper published in Nature on October 7 describes a two-stage procedure the authors call 'byteification,' for converting existing subword-based language models into byte-level models. The retrofit reportedly costs 'less than 1% of a typical pretraining budget (49.1B tokens).'

The headline comparison: Bolmo 7B, the retrofitted model, posts 'a +16.5% absolute improvement in STEM tasks over BLT 7B,' a prior byte-level model trained from scratch. The authors — Benjamin Minixhofer and colleagues including Luke Zettlemoyer, Noah A. Smith and Luca Soldaini — argue the method 'approach[es] the capabilities of subword-based systems' while targeting domains 'requiring fine-grained textual understanding.'

Architecturally, the paper describes 'non-causal boundary prediction with one byte of lookahead during prefill,' pitched as a fix for a mismatch between subword tokenizer expressivity and earlier byte-level approaches. The authors also claim post-trained subword models can enhance byteified versions 'without any extra training cost' via task arithmetic. The abstract publishes no per-task numbers beyond the STEM comparison with BLT 7B. Two researchers we track shared the paper's link.