InertiaKV: temporal aggregation matters more than KV scorer
TL;DR
- InertiaKV-Lazy, a periodic-refresh EMA variant, reports 1.34-1.46x decode throughput relative to full-refresh InertiaKV.
- Under EMA aggregation at aggressive compression, order-preserving scorer changes yield nearly identical eviction sets; KeyDiff, key norm, recency and a learned scorer degrade quality.
- A score-free variant that ranks context once and freezes it costs an average +0.03 quality delta across LongBench, LongBench-v2 and RULER.
An EMA-based eviction method with periodic refresh, InertiaKV-Lazy, "yields 1.34-1.46× decode throughput relative to full refresh InertiaKV," according to a new arxiv paper that two experts in our Who's Who directory have already shared. The point isn't the speedup itself. It's where it comes from: not a smarter token scorer, but a simpler temporal rule.
Bo Zeng and co-authors report that under aggressive KV compression, exponential-moving-average aggregation makes "approximately order-preserving scorer modifications largely indistinguishable at the eviction-set level." Value-norm and entropy variants stay close to attention. KeyDiff, key norm, recency and a learned scorer alter the ranking and, the abstract says, "degrade substantially."
A score-free variant that ranks context once at initialization and then freezes it comes in at an average quality change of +0.03 across LongBench, LongBench-v2 and RULER.
Shared on Bluesky by 2 AI experts
Originally reported by arxiv.org
Read the original article →Original headline: What Matters for Aggressive Decoding-Time KV Eviction? Temporal Aggregation and Ranking Preservation