Sparse Delta Memory scales linear RNN state via sparsity

Linear-attention models are the efficiency-first cousins of transformers, and their pitch has always been the same one, a fixed state size and a fixed amount of compute per token. The catch, according to a new arXiv preprint introducing Sparse Delta Memory, is that keeping the state small enough to stay cheap is exactly what makes these models fall behind on long-context recall. Grow the state and you claw the recall back, but you also pay in FLOPs. That trade is the wall the paper is trying to break.

The method, SDM, extends the Gated DeltaNet architecture by swapping the dense key-value outer product for sparse reads and writes to a large explicit memory. The claim is that this scales the hidden state 'to orders of magnitude higher capacity' without the FLOP penalty a naive state-size increase would carry. Under an isoFLOP constraint and with an identical number of parameters, the authors report that a higher state memory capacity 'significantly improves performance on in-context learning and long-context retrieval tasks'.

The second half of the paper is arguably the more interesting bit. By learning the initial state of the SDM memory, the same buffer doubles as parametric memory the model can consult at inference, and the authors say this further lifts scores on 'a wide range of common-knowledge and reasoning tasks'. It is a small architectural move with a big functional consequence, one buffer serving both in-context recall and baked-in knowledge.

Take the specifics as reported, not settled. The preprint is dated 8 July 2026, and the abstract does not commit to concrete benchmark numbers, model sizes, or baselines beyond Gated DeltaNet. What the write-up does not give you is a clean picture of how SDM stacks up against softmax transformers at matched scale, whether the sparse routing holds up under adversarial long-context stress, or the wall-clock cost of the memory operations on real accelerators.

If the isoFLOP claim survives independent replication, this is the kind of result that keeps the linear-RNN branch of the tree alive as a serious alternative to transformer-only scaling. The teams to watch are the ones already shipping DeltaNet-family models.