The RNN Homunculus: Learning to Gate Long-Term Memory
2 directory members surfaced this signal.
“I really enjoy finding gaps in our understanding of text-book deep learning. The RNN Homunculus: Long-range input vanishes RNN gradients. Gates (GRU, LSTM) help, but gates are themselves an RNN. Ie: gates are not sufficient. We need short-range data to lear…”