Bolmo 7B byte-level LLM tops BLT by 16.5pt on STEM tasks

Bolmo 7B, a retrofitted byte-level language model, gains 16.5 percentage points on STEM tasks over BLT 7B, a byte-level baseline. The training budget: 49.1 billion tokens.

The method, which authors Benjamin Minixhofer, Luca Soldaini and colleagues call "byteification," converts an existing subword language model into one that operates on raw bytes. "Models that instead operate directly on the byte encoding of text avoid these limitations, but until now they have lagged behind subword-based models," the paper reports in Nature. Byteification, as the authors put it, "establishes a connection between existing subword-level LLMs and byte-level LLMs."

Training splits across two stages. The first, 9.8 billion tokens of distillation, keeps the global model frozen while learning encoder, decoder and boundary-predictor weights. The second, 39.3 billion tokens, trains end-to-end. The boundary predictor, which decides where to cut the byte stream into patches, reaches above 99% accuracy by exploiting the fact that "subword tokenizers use information about future bytes to place token boundaries."

The parameter overhead is small. Bolmo 7B adds about 4.5% to its source model, roughly 330 million more parameters. A larger retrofit, Bwen 8B, outperformed every comparable publicly available byte-level model on the authors' benchmarks. Two researchers on our radar had shared the paper within days of its 7 October publication.