TTR-Compressor Matches MLP Merger With 55.6% Fewer Visual Tokens

TTR-Compressor, a sequence compression operator from researchers at the Chinese University of Hong Kong, Token Foundry and Alibaba Group, matches the widely used MLP visual-token merger while using 55.6% fewer visual tokens, and in its convolutional variant beats the strongest attention-based compressor on RULER retrieval tasks by 12.24 points.

The paper on Hugging Face reframes sequence compression — the step that collapses a block of tokens into one summary vector — as a test-time regression problem. Input tokens define a ridge-regression system; its closed-form coefficients become the compressed representation; the regression itself is learned end-to-end by backpropagation. The authors argue existing operators share "a fundamental structural limitation": they "aggregate tokens using weights that do not account for inter-token relationships." Mean pooling uses uniform weights, projection uses fixed position-dependent weights, attention-weighted pooling scores tokens independently. The consequence, the paper writes, is that "redundant tokens can reinforce one another, while distinctive information is diluted."

For visual-token merging the authors swap the standard MLP merger inside a Qwen3-VL ViT + Qwen3-4B-Instruct stack. Under 2×2 merging the average score rises from 54.93 to 55.92, and the convolutional variant reaches 56.03. Under more aggressive 3×3 merging the gains concentrate on text-rich benchmarks: +4.10 on OCRBench, +4.54 on InfoVQA, +3.56 on ChartQA, for +2.58 on average. The 3×3 convolutional variant averages 54.85, close to the MLP merger's 54.93 at 2×2 while using fewer than half the visual tokens.

The compressor adds only 2d trainable parameters per block via a T×T Cholesky solve. At a larger 1B-parameter / 100B-token setting the authors report 0.7996 average RULER accuracy against 0.735 for attention-weighted pooling, with the gap widening at longer context lengths. It is one of 98 multimodal stories we have tracked in the last 90 days.