HiLS Attention Extrapolates 64× at 90% Retrieval Accuracy

The pitch tucked inside a new arXiv paper is that you no longer have to trade accuracy for context length on the attention side, at least according to its authors. They call the method Hierarchical Landmark Sparse Attention, HiLS for short, and they claim it matches or in some cases beats full attention at in-domain lengths while extrapolating past 64× the training context with 90% retrieval accuracy.

The mechanism, as described in the abstract, is chunk-wise sparse attention where the chunk selection is learned end-to-end under the language-modeling loss rather than picked by a hand-designed heuristic. Each query attends independently to each retrieved chunk, and the outputs are fused according to chunk retrieval scores. Because those scores sit inside the forward attention computation, they get gradient signal from the LM loss directly, which is the piece the authors point at as the reason earlier chunk-wise sparse-attention methods fell short of full attention.

The part that matters for anyone running a production stack is the conversion story. The paper claims existing full-attention models can be moved onto HiLS with lightweight continued pretraining while preserving in-domain performance. If that holds, ultra-long context stops being a from-scratch retrain and becomes a fine-tune line item, which is a very different budget conversation.

The honest caveat is that this is an arXiv preprint reporting the authors' own numbers. The abstract does not spell out which base model produced the 64× headline, what the training context length actually was, how demanding the continued pretraining really is, or how the method holds up on tasks that require dense reasoning across many chunks rather than retrieval-style lookups. Take the specifics as reported, not settled.

If independent teams reproduce even a fraction of the claim, the near-term beneficiaries are the open-weights labs and inference providers that have been paying quadratic KV costs to serve long windows. Watch for third-party conversions on well-known checkpoints in the coming weeks.