Yao co-authors FALCON, unifying SSMs and fast-weight memory

Eleven researchers, including Andrew Chi-Chih Yao, have posted a paper on arXiv arguing that recurrent fast-weight memories and selective state-space models are the same object: an online learning rule that updates a fixed-size recurrent state as each token arrives.

The paper, titled 'Fast Weight Attention for Continual Learning,' derives normalized first-order updates for squared-error regression as a family of three: 'Falcon-1 (a scalar NLMS update), Falcon-2 (its per-column extension), and Falcon-3 (a sliding-window mini-batch update).' Each has an inner-product analog (Falcon-1A/2A/3A), and the authors publish recurrent, masked-parallel, and chunk-parallel implementations with what they call 'numerically stable positive-decay renormalization.'

The empirical bar is modest. Representative variants 'remain competitive in language modeling and improve length extrapolation on variable-digit addition,' the abstract reports. It names no baselines and publishes no benchmark numbers.

The framing is what stakes the paper. If the derivation holds, selective state-space models and fast-weight networks are two parameterizations of the same regression step, one the authors say 'separates temporal alignment, plasticity, forgetting, and bounded rehearsal in recurrent sequence models.'