Paper: https://t.co/IoDcRINaAL This is one of a whole bunch of recent papers reviving study of recurrent neural networks. One weird omission is not testing LSTM RNNs. Surely they remain the canonical successful RNN architecture? Another completely uninvestigated thing is the
AI Weekly's analysis
→
- Supervised Memory Training reduces RNN pretraining to supervised learning on one-step memory transitions, enabling time-parallel training without unrolling.
- A Transformer encoder trained on a predictive state objective supplies the memory labels the RNN then learns to reproduce.
- The authors report SMT beats standard backpropagation through time on language modeling and pixel sequence modeling.
Read full analysis →
