Byteification retrofits LLMs to match subword benchmarks

A new Nature paper shows that major open-weights language models can be retrofitted to read raw bytes instead of subword tokens, closing a performance gap that has held byte-level systems back for years, at a fraction of normal training cost.

Benjamin Minixhofer and co-authors call the method 'byteification' and apply it to OLMo 2 1B, OLMo 3 7B, Qwen3 8B and Llama 3 8B, producing byte-level variants named Bolmo, Bwen and Blama. The authors state the gap they are closing plainly: 'Models that instead operate directly on the byte encoding of text avoid these limitations, but until now they have lagged behind subword-based models in performance.'

Their two-stage conversion uses 9.8 billion tokens in stage one and 39.3 billion in stage two, which the paper puts at less than 1% of what a typical pretraining run consumes. The headline comparison: Bwen 8B, converted from Qwen3 8B, outperformed the earlier byte-level BLT 7B by 16.5 absolute percentage points on STEM tasks within a 40-benchmark evaluation suite. Parameter overhead relative to source models stayed within a narrow band: Bolmo 1B 0.7% smaller, Bolmo 7B 4.5% larger, Blama 8B 2.7% larger, Bwen 8B 1.5% larger. That leaves the surrounding subword ecosystem (instruction tuning, task-arithmetic merges, serving tooling) usable with little rework.

Why bytes at all. Subword tokenization chops rare words, non-English scripts, DNA bases and code symbols into fragments, and that costs accuracy where single characters carry meaning. The authors frame the payoff this way: 'Our results remove a long-standing performance barrier to end-to-end byte-level language modelling, demonstrating that models operating on raw text encodings can scale competitively while offering advantages in domains requiring fine-grained textual understanding.'

The paper reached two researchers we track within its first day online. It does not benchmark against closed frontier systems, and the published figures do not include inference-cost comparisons at production serving scale.