Byteification retrofits Llama 3, Qwen3 to byte-level models

A new Nature paper describes "byteification," a retrofit that converts existing subword language models to operate directly on raw UTF-8 bytes, using roughly 49.1 billion training tokens — a budget the authors put at "less than 1% of a typical pretraining budget."

The method is applied to four open-weight models and the retrofits are rebranded accordingly: Bolmo 7B (from Olmo 3 7B), Bolmo 1B (from OLMo 2 1B), Bwen 8B (from Qwen3 8B) and Blama 8B (from Llama 3 8B). The authors report Bolmo 7B delivers a "+16.5% absolute improvement in STEM tasks over BLT 7B," a prior byte-level baseline, and that Bwen 8B outperformed all comparable byte-level models they tested. Two of the researchers in our Who's Who directory shared the paper's link.

"Our results remove a long-standing performance barrier to end-to-end byte-level language modelling," the authors write, "demonstrating that models operating on raw text encodings can scale competitively." They also frame the method's reach modestly: "Byteification lets us train byte-level LLMs with a performance close to that of state-of-the-art subword-level LLMs" — close to, not past.

The retrofit splits across two stages, 9.8 billion tokens in the first and 39.3 billion in the second. Per-token inference cost against the subword source models is not reported in the abstract.