Kumar, Isola propose SMT to pretrain RNNs without recurrence

Akarsh Kumar and Phillip Isola describe a training recipe that pretrains recurrent networks without ever unrolling them through time. Their method, Supervised Memory Training, reduces RNN pretraining to supervised learning on one-step memory transitions of the form (m_t, x_{t+1}) to m_{t+1}, so the gradient path between any two tokens is a stable O(1) rather than a chain that has to walk the whole sequence.

The memory labels themselves come from a separate Transformer-based encoder trained on what the authors call a predictive state objective, "retaining only information from the past necessary to predict the future." That split, one network deciding what to remember and another learning how to update memory, is what lets the RNN training loop run in parallel across time. Standard backpropagation through time, the authors write, "is sequential in time, limiting parallelism, and suffers from vanishing or exploding gradients, making long-range associations difficult to learn."

They report that SMT "outperforms BPTT when pretraining various RNN architectures on tasks like language modeling and pixel sequence modeling." The abstract names no specific architectures, parameter counts, or per-task scores.