C²KV Framework Claims 17× Long-Context LLM Inference Speedup

The cost of running a long-context LLM at scale is dominated by the KV cache: keep it around and memory explodes, throw it away and you re-encode the same documents on every request. A new ACM SIGKDD 2026 paper, C²KV on arXiv, argues you can have it both ways, and claims up to a 17× inference speedup on long contexts while preserving generation quality.

The pitch is worth unpacking. Instead of only reusing KV caches that share a common prefix, which is how most production systems handle repeated system prompts, the authors describe a lightweight sidecar Extractor with learnable compression tokens and a structured attention flow. The framework produces what the paper calls a composable and compressed KV cache manifold that is explicitly designed to be position-agnostic, and that is the property that lets pieces of cache be stitched together across requests rather than only reused when they sit at the start of the prompt. The base model itself is not modified, and a compression-concatenation co-training strategy is used to align how KV chunks are extracted with how they will later be recombined downstream.

If the numbers hold, the practical implication is straightforward. Retrieval-augmented generation, multi-document reasoning, and any workload where the same documents get pulled into many different prompts have been paying to re-encode the same tokens over and over. A reusable, compressed KV representation shrinks that cost, and because the technique is bolted on rather than baked into the model, self-hosted inference stacks can in principle adopt it without a retraining cycle. That is a meaningful lever for anyone running open-model RAG at volume, and it is why inference-serving vendors will care.

The honest caveat is that this is a research claim, not a production result. '17×' is an 'up to' number, and the abstract does not specify which base models, context lengths, or workloads produced that peak, nor how much quality is really 'preserved' on adversarial retrieval or fine-grained citation tasks. Teams tied to hosted APIs also cannot adopt a sidecar approach that needs model-adjacent access, so the benefit skews toward the self-hosting side of the market.

Still, the direction is the interesting part. Non-prefix KV reuse has been the obvious next frontier past prompt caching, and a peer-reviewed SIGKDD result that puts a specific speedup number on the table gives the open-model inference stack something concrete to try to reproduce.