RecSA stretches Qwen3-4B to 2M tokens at logarithmic cost

A new arxiv preprint from Ruitian Wang, Xinyu Yang, Minshen Zhang and Hao Zhang claims that a 4B-parameter model trained only at 32K tokens can be stretched to 2M-token context at logarithmic attention cost, and beat dense baselines on a standard long-context benchmark while doing it.

The method, which the authors call RecSA, "builds a tree over the KV cache in which each node summarizes its subtree with one key and one mean-pooled value." Keys come from one pooler per layer, shared across levels and distilled from dense attention. Each level selects at most k candidates, so a query pays O(log L) selection cost and reads O(log L) entries. The trick for length extrapolation is positional: "Each entry's RoPE position is its index among the candidates of its level, so at any length every relative distance stays within the range seen during training."

The headline number is on RULER. Initialized from Qwen3-4B and trained on 0.62B tokens at the base model's 32K length, RecSA "outperforms all baselines on RULER at every length from 131K to 2M; at 1M, it scores 60.5, compared with 40.0 for the best baseline, Dense + DCA." The authors also report wall-clock wins: "at 524K on one H200 GPU, it accelerates prefill by 3.2x and decoding by 3.8x."

The paper itself is terse about what it doesn't test. The abstract reports only RULER, not open-ended long-document reasoning, and only Qwen3-4B as a base. The speedups are tied to the authors' own optimized kernels. Two researchers we follow circulated the preprint the week it dropped, which is a modest signal that the hierarchical-selection line of work is still finding new room.