DeepMem byte model beats matched Transformer at 100M scale

DeepMem, a byte-level language model, reports 1.0134 bits-per-byte on FineWeb-Edu at roughly 100 million parameters, against 1.0457 for a matched raw-byte Transformer and 1.0187 for a FastWeight baseline. Afshin Khadangi, the paper's single author, posted it to Hugging Face this week under the title "Going Deeper With Memories."

The design splits the sequence into two parallel streams. A learned semantic stream is progressively shortened; alongside it runs what the paper calls "a lossless carrier and explicit detail sidebands preserve byte identity" through an invertible subband transform. Fast-weight memory blocks handle the encoding. At the base, the sequence narrows to 16 positions, where a shared state is refined twice. That bottleneck step, the abstract says, requires "approximately 7,600× fewer counted FLOPs than equivalent full resolution reasoning." An attention-based super-resolution decoder then restores byte-level representations.

The gain carries beyond FineWeb-Edu. On PG19, DeepMem reaches 1.4402 BPB versus 1.4635 for the Transformer; across eight FLORES languages, it posts a 1.9694 macro-average against 2.0082, "with a modest tradeoff on Stack-smol code." On HellaSwag, PIQA and ARC, and against repeated-character, Antspeak, uppercase and random-case perturbations, the paper reports parity rather than gain.

Training ran on four NVIDIA Tesla V100-SXM2 GPUs over 50,000 optimizer steps, with a 2,048-byte context across three random seeds and roughly 26.2 billion byte presentations. The baselines section lists BLT and AU-Net alongside the Transformer and FastWeight, though the abstract's headline numbers compare only to the latter two. The work lands inside a steady run of tokenizer-free and alternative-attention papers the tracker has logged this quarter.