LoopVQ compresses looped transformer KV cache to 0.5 bits

A looped transformer's key-value cache grows linearly with its loop depth, so memory, not parameters, is the architecture's main serving bottleneck. The paper's own framing: each loop step "still stores its own keys and values," and the cache "becomes the main memory bottleneck" even though the weights do not.

An arXiv preprint from Minsoo Cheong and Sungjoo Yoo attacks that bottleneck through quantization. They observe that "the KV cache of looped models is highly redundant along the loop axis" and that loop steps differ substantially in their sensitivity to quantization. Their method, LoopVQ, jointly vector-quantizes channels across multiple loop steps and fine-tunes the codebooks to match the full-precision model.

The headline figure is 0.5 bits per element, at which LoopVQ beats existing KV-cache vector-quantization methods designed for standard LLMs on general tasks, long chain-of-thought reasoning, and long-context retrieval, evaluated on the Ouro and Huginn models. Stacked with TaSQ, the authors write that it "retains much of the BF16 accuracy even at 0.25 bits per element."

On the serving side the numbers get concrete. Dropped into SGLang, LoopVQ "nearly halves the decode latency at a 32K-token context" and "raises the peak serving throughput by up to 4.3× and 6.1× on the 1.4B and 2.6B Ouro models" versus the BF16 baseline. The abstract reports no per-task accuracy numbers to sit alongside those speedups.