Paper: tuning <2% of heads lifts Qwen3-VL by 6.03 points

Fewer than 2% of attention heads do the work. That is the finding from "A Broader Look into MLLM's Retrieval Heads," posted to arXiv on October 23, 2026, by Jia Wang, Yuqing Li, Jiangnan Li, Zheng Lin and Mo Yu from the Institute of Information Engineering, Chinese Academy of Sciences, and Tencent. The team looked across 18 model checkpoints spanning language, vision-language, and omni-modal architectures.

The authors call these "retrieval heads," the sparse attention mechanisms that move information from a long context to the output. Across modalities, "retrieval scores concentrate in a small set of heads for both spoken and displayed evidence," the paper reports. Ablating the subset shared across modalities, which the authors describe as "fewer than 2% of all heads," degrades document understanding, spatial perception, and visual mathematics. Ablating retrieval heads more broadly eliminates "nearly all audio-visual retrieval."

The team then trained those heads with a ranking objective. On MMDocIR, Qwen3-VL-8B's page Recall@1 rose by 6.03 points, and the approach improved document retrieval across three model families.

The paper is listed as work in progress.