Simple beats complicated: We show that switching to a sliding-window attention mask with attention sinks (at no cost) beats linear attention post-training. Huge thanks to my collaborators Rhea Sukthanker, Pashmina Cameron, and Emy Gervais. Paper: arxiv.org/abs/2608.28444
AI Weekly's analysis
→
- On the long-context reasoning tasks the paper cites (Needle-in-a-Haystack and BABILong), SWA scored 2 to 10 times higher than post-trained linear attention.
- The authors argue linear-attention retrofits have not been properly compared to simpler baselines, and that SWA with sinks needs no post-training at all.
- Their bottom-line recommendation is to switch to SWA rather than continue post-training linear models for inference memory savings.
Read full analysis →
