Triton kernel cuts relative-position attention memory up to 128×

At 8K-token sequences, a new Triton kernel reports forward-pass speedups of 1.8 to 6.4× and peak memory reductions of 2.9 to 128× over the strongest baseline for each relative-position attention mechanism it tested, according to a preprint on arXiv by Delyan Boychev, Ivo Petrov and Dimitar I. Dimitrov. The backward pass runs 2.0 to 21× faster with 1.4 to 54× less peak memory.

Relative-position attention keeps showing up as the RoPE alternative in audio and other structure-sensitive stacks. The infrastructure has been the problem. "Unlike RoPE, which encodes position via per-token rotations directly on the queries and keys, relative-position terms generally require pairwise offset-dependent computation or lookups, complicating their integration into fused attention kernels and introducing severe memory, bandwidth, and compute overheads as sequences scale," the authors write.

Their fix folds additive relative-position terms into FlashAttention-style tiled computation using three moves: offset-aware tiling to avoid redundant indexing, reused key/query-relation products across token pairs in forward and backward passes, and offset-based gradient sharding to eliminate atomic contention. Downstream, the paper reports long-context end-to-end transcription that is "1.8× faster and uses 2.4× less memory." The abstract names no specific mechanism at the 128× ceiling and no specific one at the 2.9× floor.