arxiv.org web signal

LightningMoBA Cuts Long-Context Gen Time 1.2-1.6x vs FlashMoBA

TL;DR

  • LightningMoBA accelerates end-to-end generation by 1.20 to 1.60x over FlashMoBA at matched Top-16 retention across five checkpoints.
  • The system swaps fixed Top-k selection for dynamic routing that thresholds per-query block scores and emits variable-cardinality sparse metadata.
  • Mean RULER long-context score rises from 74.42 to 75.28, so the speedup comes without a quality trade-off in the abstract's numbers.

LightningMoBA, a new arXiv paper from Yingqi Shi, Junru Lu, Yun Liu, Di Yin and Xing Sun, reports 1.20 to 1.60x faster end-to-end generation than FlashMoBA at matched Top-16 retention across five checkpoints, while the mean RULER long-context score rises from 74.42 to 75.28.

The target is a specific bottleneck. FlashMoBA, the fast implementation of Mixture of Block Attention, "retains a fixed Top-k budget and query-centric schedule," which the authors say applies one budget across different score distributions and fragments K/V reuse when several ranges select the same column. LightningMoBA swaps all three pieces at once. Dynamic routing thresholds each query's block scores and "directly emits variable-cardinality sparse metadata." KV-centric ownership "groups selected rows by (query head, KV block) column, splits high-fanout columns into bounded tasks, and reuses each task's loaded KV tile." And on Hopper, "pipelining overlaps regular K/V transfers and irregular query gathers with Tensor Core computation."

The abstract reports one retention setting (Top-16) and one benchmark (RULER); numbers at other budgets, memory cost against FlashMoBA, and the code release path are not stated. It lands into a busy stretch for sparse-attention kernel work on our inference desk, which has logged 145 stories over the last 90 days.