Nature paper retrofits LLMs to run on raw bytes at ~1% compute

"Byteification" is the name the authors give to a conversion procedure that turns an existing subword language model into one that reads raw bytes, and the Nature paper reports it works at under 1% of a typical pretraining compute budget. The team, led from the Allen Institute for AI with collaborators at the University of Washington, Cambridge and Edinburgh, applies the recipe to three open models they call Blama 8B, Bwen 8B and Bolmo 7B.

The reported headline gain is a "16.5% absolute improvement over previous byte-level approaches on STEM tasks," while the byteified models' overall benchmark scores approach those of the subword source models they were built from. The abstract situates the method against the status quo bluntly: "Recent advances in artificial intelligence (AI) have largely been driven by large language models, deep neural networks that operate over discrete units called tokens."

The architecture keeps the global transformer from the source model and wraps it in local mLSTM encoder/decoder layers plus a non-causal boundary predictor that looks ahead by a single byte. Training runs in two stages for roughly 49.1B tokens total: 9.8B first to recover subword behavior, then 39.3B for end-to-end optimization. Two researchers in our tracker flagged the paper.

The comparison baselines are prior byte-level methods rather than frontier subword SOTA, and the three retrofitted checkpoints sit in the 7B to 8B range; the abstract publishes no latency or inference-cost numbers for the mLSTM wrappers at serving time.