Olmo 3 7B retrofit to byte-level beats BLT by 16.5% on STEM

A byte-level variant of Olmo 3 7B, built via a two-stage retrofit the authors call "byteification," posted a +16.5% absolute improvement on STEM tasks over BLT 7B while using less than 1% of a typical pretraining budget: 49.1B tokens.

The method is set out in a Nature paper by Benjamin Minixhofer, Tyler Murray, Luke Zettlemoyer, Luca Soldaini, Valentin Hofmann and colleagues. They converted four open-weight subword models (OLMo 2 1B, Olmo 3 7B, Qwen3 8B Base, Llama 3 8B) into byte-level versions named Bolmo 1B, Bolmo 7B, Bwen 8B and Blama 8B. Added parameters ranged from −0.7% for Bolmo 1B to +4.5% for Bolmo 7B. Stage 1 training used 9.8B tokens; Stage 2 used 39.3B. Boundary prediction is non-causal, looking ahead "1 byte of future context."

"Byteification lets us train byte-level LLMs with performance close to state-of-the-art subword-level LLMs," the authors write. On character-level benchmarks such as CUTE and EXECUTE, long a known weakness of subword tokenization, "the byteified models substantially surpassed their subword-level counterpart."

The abstract reports aggregate gains, not per-task throughput or inference latency. The experiments cap at 8B parameters, so a frontier-scale retrofit is not something the paper settles.