sliding window beats linear attention
Sliding-window beats linear attention
4 experts across 3 network communities independently surfaced this.
4 experts
3 communities
1 sources clustered
How the network is reacting
3 experts are independently emphasizing research & technical analysis.
Reaction converging
Research & technical analysis
3 expertsEvidence, methods and technical implications.
“Huge thanks to my collaborators @RheaSukthanker, @CameronPashmina, and @Emy_Aze. Paper: https://arxiv.org/abs/2608.28444”
2 experts discussed this · 2 posts
Alexia Jolicoeur-Martineau: Simple beats complicated: We show that switching to a sliding-window attention mask with attention sinks (at no cost) beats linear attention post-training. Huge thanks to my collaborators Rhea Sukt…
Miguel Alonso Jr.: Simple beats complicated: We show that switching to a sliding-window attention mask with attention sinks (at no cost) beats linear attention post-training. Huge thanks to my collaborators Rhea Sukt…
James MacGlashan: Makes me wonder if you can remove the sinks if you use softmax-1?
Open the full discussion →