Pretraining Recurrent Networks without Recurrence
4 experts across 3 network communities independently surfaced this.
“Paper: https://t.co/IoDcRINaAL This is one of a whole bunch of recent papers reviving study of recurrent neural networks. One weird omission is not testing LSTM RNNs. Surely they remain the canonical successful RNN architecture? Another completely uninvesti…” evidence ↗
Research & technical analysis
2 expertsEvidence, methods and technical implications.
“Pretraining Recurrent Networks without Recurrence It sidesteps the limitations of RNNs by using a Transformer teacher to learn strong predictive state representations, then uses supervised learning to train the memory transition function. arxiv.org/abs/2606…”