'RNN Homunculus' paper: gated RNNs can match Mamba on long memory

Gated RNNs, trained the right way, can match or beat Mamba and xLSTM on long-memory tasks. That is the claim in a new arxiv preprint by Zeyu Fan, Alexandru-Dragos Manolache and Jan van Gemert.

The authors look at why gated recurrent networks like GRUs and LSTMs are supposed to handle long-range dependencies and often don't. The standard story is that gates open additive copy paths through time. The paper says that story is 'correct, but incomplete': the gates themselves have to be learned, and learning them runs into the same vanishing-gradient problem the gates were meant to solve. 'Gated architectures struggle when all training samples require long-range memory and thus equally suffer from the vanishing gradient problem,' the paper reports.

Their fix is unglamorous. The authors mix short-dependency samples into training alongside the long ones, which lets gated models 'learn solutions that capture long-dependency samples, but also generalize across different class labels.' With that curriculum, they write, 'gated RNNs match or even exceed the test accuracy of modern architectures like Mamba and xLSTM.' The experiments are controlled sequence classification tasks, not language modeling, so the direct comparison to Mamba in its best-known setting is not what the paper claims to settle.