LowRAM Claims 26.9× KV Cache Compression at Test Time
TL;DR
- LowRAM reports 26.9× KV cache compression at 94.5% of baseline performance on long contexts when stacked with quantisation and eviction.
- The method fits a lightweight per-head predictor at test time that reconstructs values from keys, storing residuals only for the worst-reconstructed tokens.
- At 4× compression the authors claim near-lossless behaviour across three model families and up to 5× higher peak decode throughput.
LowRAM, a new test-time method for shrinking the KV cache, reports 26.9× compression at 94.5% of baseline performance on long contexts, according to a preprint posted to arXiv.
The approach treats keys and values asymmetrically. Authors Teresa Delgado, Hugo Hazard, Martin Benfeghoul, Jun Wang and Zafeirios Fountas identify what they call a complementary axis: "within each attention head, values are largely predictable from their corresponding keys." They fit a lightweight per-head predictor at test time and store residuals only for the tokens it reconstructs worst, which the paper describes as giving "4× near-lossless value compression." That is paired with cross-layer low-rank key compression.
LowRAM, the paper says, "stays close to lossless across three model families" at 4× compression, and can reach "up to 5× higher peak decode throughput" at that same ratio. The 26.9× headline figure is reached only when the method is stacked on top of quantisation and eviction, which the authors position as composable rather than competing techniques.
The abstract names no specific models, no benchmarks and no per-task scores.
Shared on Bluesky by 1 AI expert
Originally reported by paper
Read the original article →Original headline: LowRAM Hits 26.9× KV Cache Compression at 94.5% Score—No Weight Changes Required