Olmo, Qwen, Llama get byte-level retrofits in Nature paper

A Nature paper published 7 October describes a two-stage conversion procedure that retrofits existing subword language models to operate over raw bytes. Total training cost: 49.1B tokens, what the authors call "less than 1% of a typical pretraining budget."

The method produces four retrofitted models: Bolmo 7B from Olmo 3 7B, Bolmo 1B from OLMo 2 1B, Bwen 8B from Qwen3 8B, and Blama 8B from Llama 3 8B. The conversion adds 330M parameters to Bolmo 7B (+4.5%) and 120M to Bwen 8B (+1.5%). The authors report Bolmo 7B achieved a "+16.5% absolute improvement in STEM tasks over BLT 7B," and that their byteified models beat "all earlier publicly available byte-level LLMs of comparable size."

The paper names what it is trying to fix: "Subword tokenization obscures fine-grained information, which is problematic, especially for scientific data." Evaluation spans HumanEval, MBPP, GSM8K, MMLU, ARC and the character-understanding tasks CUTE and EXECUTE.

Training splits 9.8B tokens in stage one, 39.3B in stage two. The paper's inference-speed claim is phrased as "competitive," not benchmarked in the abstract against named production stacks.