PACE 4-Bit State Quantizer Matches 8-Bit DAMP at 2/3 Memory

A new post-hoc quantizer rounds the recurrent state of linear-attention language models to four bits by rounding it less often: once every eight generated tokens instead of after every token, which is where prior four-bit methods lost accuracy. The method is called PACE, and the paper went up on Hugging Face on 22 October.

The setup matters because at 64 concurrent sequences 'the fp32 states take between 10 and 41 % of the memory that weights and states occupy together,' and shaving that lets more sequences share an accelerator. Prior four-bit quantizers either fine-tune part of the weights or lose accuracy; the eight-bit alternative, DAMP, 'reports 9.9 bits per value' and leaves memory on the table.

PACE borrows from how chunked kernels already behave during prefill. 'During prefill, chunked kernels process 64 to 1,024 tokens at a time and write the state only at the end of each chunk, so every published quantizer already rounds the prefill state once per chunk,' the authors write. They extend the pattern into decoding by holding recent keys and values in a buffer and advancing the state once per flush. PACE also finds that large entries of the gated-delta state 'gather in columns' and shares a quantization scale per column; PACE-4/8 keeps the quarter of heads in each layer that retain rounding error longest at eight bits.

On four hybrid backbones, Qwen3.5-4B, 9B, 27B and Qwen3.6-35B-A3B, PACE-4/8 is 'competitive with eight-bit DAMP on the three larger models at about two thirds of its memory,' and 'PACE with four-bit heads throughout scores higher than Q-Mamba at equal memory.' On the smallest 4B backbone its average is lower than DAMP INT8, with 'lower scores on GSM8K, MMLU-Pro and IFEval.' None of the structure PACE relies on 'is shown to carry over to other model families, to models whose layers are all recurrent, or to token mixers with a different update or decay.' Throughput was measured only on Qwen3.5-4B on one H100 PCIe GPU at a 512-token context, with Triton kernels released for periodic decoding over a packed four-bit state. It arrives in the middle of a busy week for inference research on our tracker.