sliding window beats linear attention
4 experts across 3 network communities independently surfaced this.
4 experts
3 communities
1 sources clustered
Research & technical analysis
3 experts
Evidence, methods and technical implications.
“Huge thanks to my collaborators @RheaSukthanker, @CameronPashmina, and @Emy_Aze. Paper: https://arxiv.org/abs/2608.28444”
2 experts discussed this · 2 posts
Alexia Jolicoeur-Martineau: Simple beats complicated: We show that switching to a sliding-window attention mask with attention sinks (at no cost) beats linear attention post-training. Huge thanks to my collaborators Rhea Sukt…
Miguel Alonso Jr.: Simple beats complicated: We show that switching to a sliding-window attention mask with attention sinks (at no cost) beats linear attention post-training. Huge thanks to my collaborators Rhea Sukt…
James MacGlashan: Makes me wonder if you can remove the sinks if you use softmax-1?
Open the full discussion →
3 experts across 2 network communities independently surfaced this.
3 experts
2 communities
1 sources clustered
Research & technical analysis
2 experts
Evidence, methods and technical implications.
“https://arxiv.org/abs/2608.17981 Recurrent transformer that injects activations from the top layers to the bottom layers at the next step (https://arxiv.org/abs/2608.08888). Why does this work without training? — via @rosinality https://x.com/rosinality/sta…”
Policy & governance
1 expert
Rules, institutions and accountability.
“Full-bandwidth transformer Xi Wang, Ziyang Cai, Zheng Zhan, Harry Dong, Ying Fan, Gustavo de Rosa, Tim Pearce, John Langford https://t.co/FUDCgqgkqW [𝚌𝚜.𝙰𝙸] https://t.co/eiA0jqzYAQ”
3 experts discussed this · 5 posts
Sung Kim: At decoding time, when you feed **previous hidden state** into the input together with token embedding, and it boosts performance for free. It unlocks the ** full bandwidth ** of the transformer: -…
Sung Kim: - It exposes past information across *depth* to the current layer/position Paper: Full-bandwidth transformer ( arxiv.org/abs/2608.08888 )
SE Gyges: u like can't backprop this tho because your path length increases without limit
Open the full discussion →
2 directory members surfaced this signal.
2 experts
2 communities
1 sources clustered
“arXiv: https://arxiv.org/abs/2608.25832 alphaXiv: https://alphaxiv.org/abs/2608.25832 HF Paper: https://huggingface.co/papers/2608.25832 Code: https://github.com/TextArena/TextArena”
“Bobby Cheng, Adam Gaber, Zhengyuan Liu, Catherine Arnett, Omer Goldman, Cheston Tan, Leshem Choshen Skill Issue: Are Skills Language-Invariant in LLMs? https://arxiv.org/abs/2608.25832”
Established
AI field signal
Signal
18d ago
2 directory members surfaced this signal.
2 experts
2 communities
1 sources clustered
Research & technical analysis
2 experts
Evidence, methods and technical implications.
“Many other details and experiments are in the paper: https://arxiv.org/pdf/2608.09703 We released our checkpoints here: https://huggingface.co/nthngdy/matryoshka-3B Many thanks to my co-author and advisor @yoavartzi for the precious guidance, and to @NVIDIA…”
matryoshka 3B model released
2 directory members surfaced this signal.
2 experts
2 communities
1 sources clustered
Research & technical analysis
2 experts
Evidence, methods and technical implications.
“Many other details and experiments are in the paper: https://arxiv.org/pdf/2608.09703 We released our checkpoints here: https://huggingface.co/nthngdy/matryoshka-3B Many thanks to my co-author and advisor @yoavartzi for the precious guidance, and to @NVIDIA…”
Established
AI field signal
Signal
18d ago
⚡ 33 h early
2 directory members surfaced this signal.
2 experts
2 communities
1 sources clustered
Research & technical analysis
1 expert
Evidence, methods and technical implications.
“https://arxiv.org/abs/2608.17981 Recurrent transformer that injects activations from the top layers to the bottom layers at the next step (https://arxiv.org/abs/2608.08888). Why does this work without training? — via @rosinality https://x.com/rosinality/sta…”
Opportunity & adoption
1 expert
New capabilities, benefits and practical upside.
“Recirculation — boost perplexity 23% with a small loop another take on RNNs for transformers, unlike looped transformers, this doesn’t loop during generation time. It only loops during prefill and the purpose is to give the model a clearer view of the input…”
Last Translation Benchmark tool
3 experts are actively discussing the implications.
2 experts
2 communities
1 sources clustered
Research & technical analysis
2 experts
Evidence, methods and technical implications.
“Last Translation Benchmark is a live paper+dataset and you can still join last-translation-benchmark.vilda.net Massive thanks to all the >250 dataset contributors and @niyatibafna.bsky.social @mukundc2k.bsky.social @maikezufle.bsky.social @pinzhen.bsky.social”
3 experts discussed this · 9 posts
Vilém Zouhar: There are many things machine translation still can't do. Help us steer the next direction by contributing hard-to-translate inputs (and be on a cool paper).
Vilém Zouhar: Multiple things made us start this effort. Typical translation benchmarks are.. ...oftentimes trivial or saturated (so they can't be used for guiding the next steps in the field) ...not evaluatable…
Vilém Zouhar: In the Last Translation Benchmark we solve both by: - collecting hard-to-translate inputs (texts, images, audios) - requiring human-readable "verification rules", which enable provable evaluation o…
Open the full discussion →
cross-lingual LLM knowledge sharing paper
1 directory member surfaced this signal.
1 expert
1 community
1 sources clustered
“arxiv.org/abs/2408.10646 arxiv.org/abs/2502.21228 And so much more (this nuance is somehow not being recognized and following the literature is so hard this days this well established branch on lack of sharing is hardly discussed outside the branch)”
ECLeKTic cross-lingual benchmark release
1 directory member surfaced this signal.
1 expert
1 community
1 sources clustered
“arxiv.org/abs/2408.10646 arxiv.org/abs/2502.21228 And so much more (this nuance is somehow not being recognized and following the literature is so hard this days this well established branch on lack of sharing is hardly discussed outside the branch)”
1 directory member surfaced this signal.
1 expert
1 community
1 sources clustered
“arXiv: https://arxiv.org/abs/2608.25832 alphaXiv: https://alphaxiv.org/abs/2608.25832 HF Paper: https://huggingface.co/papers/2608.25832 Code: https://github.com/TextArena/TextArena”
1 directory member surfaced this signal.
1 expert
1 community
1 sources clustered
“We compare to MatFormer (https://arxiv.org/abs/2310.07707), and show that Matryoshka suites offer a better size-performance tradeoff, allow more size flexibility, and deliver variable KV cache requirements In short we trade their inference-time flexibility …”
1 directory member surfaced this signal.
1 expert
1 community
1 sources clustered
“Linear attention has limited memory, so it must choose which tokens to remember and which to forget. Trying to learn this during post-training at a small cost is misguided. — x-post from @jm_alexia https://x.com/jm_alexia/status/2094414735408050687”
Established
AI field signal
Resource
8d ago
alphaXiv arxiv platform
1 directory member surfaced this signal.
1 expert
1 community
1 sources clustered
“This paper identifies that the failure starts during pretraining and that disjoint token spaces alone can cause it, even between two otherwise — x-post from @askalphaxiv https://x.com/askalphaxiv/status/2093900515205931341”
LLM skills language invariance paper
1 directory member surfaced this signal.
1 expert
1 community
1 sources clustered
“Self centered, sure, but I wonder how they compare on our games, playing against themselves or each other www.alphaxiv.org/abs/2608.ski...”
