Nature paper retrofits subword LLMs to run on raw bytes

A new Nature paper describes a procedure the authors call 'byteification,' a two-stage retrofit that converts an existing subword language model into one that reads raw bytes. The headline cost: training on 'approximately 49.1 billion tokens total (less than 1% of typical pretraining budgets).'

One converted model, Bwen 8B, posts a '+16.5% absolute improvement in STEM tasks' against BLT 7B, the byte-level baseline the paper benchmarks against. A second, Bolmo, inherits instruction-following from Olmo 3 through task arithmetic with no additional training. The authors (Benjamin Minixhofer, Tyler Murray, Luke Zettlemoyer, Noah A. Smith, Luca Soldaini, Valentin Hofmann and colleagues) argue that subword tokenization creates problems where models 'suffer from limited character-level understanding,' particularly on code and biological sequences.

The abstract reports 'practical inference speeds' but publishes no wall-clock numbers, and the headline STEM comparison runs 8B against 7B rather than a size-matched baseline. Two researchers we follow had shared the paper by the time it reached our tracker.