Byteification retrofits open LLMs to operate over bytes

"Byteification," described in a Nature paper by Benjamin Minixhofer, Luke Zettlemoyer, Noah A. Smith and co-authors, converts existing subword-based language models into byte-level models at a cost of less than 1% of a typical pretraining run.

The procedure uses 49.1 billion training tokens in total: 9.8 billion in a distillation stage, 39.3 billion in end-to-end refinement, and 75 million tokens of character-level data. Applied to several open-weights model families, it produced Bolmo 7B (from OLMo 3 7B), Bolmo 1B (from OLMo 2 1B), Bwen 8B (from Qwen3 8B Base) and Blama 8B (from Llama 3 8B).

The abstract reports that the retrofitted models "approach the capabilities of subword-based systems" while excelling at character-level reasoning tasks. On STEM, Bolmo 7B gained 16.5 percentage points in absolute terms over BLT 7B, a prior byte-level baseline, across a 40-task evaluation suite at 7B and a 22-task suite at 1B.

The architecture combines non-causal boundary prediction with one-byte lookahead, a Latent Tokenizer Language Model, and mLSTM layers for local encoding and decoding; post-training was carried over via task arithmetic. Two of the researchers in our tracker shared the link.