CAP pruning tops Wanda by up to 61% at 20% sparsity on ARC
TL;DR
- Causal Attribution Pruning identifies critical attention heads by measuring the performance drop when each head is masked during forward passes on reasoning problems.
- On ARC-Challenge at 20% sparsity, CAP reports relative accuracy gains of up to 61% over the Wanda baseline, with especially large gains on Llama-3.
- The method is training-free and tested on Llama-3-8B-Instruct and Mistral-7B-Instruct across GSM8K, StrategyQA, and ARC-Challenge, but weakens at 50% sparsity.
A new pruning method reports relative accuracy gains of up to 61% over Wanda on ARC-Challenge at 20% sparsity, according to the arxiv preprint. Causal Attribution Pruning, or CAP, identifies critical attention heads by masking each head during forward passes on a small calibration set of reasoning problems, then converts the resulting causal score into weight-level importance values for the corresponding projection matrices.
The authors frame it as a break from the dominant family of pruning criteria: 'Unlike magnitude-only or activation-based criteria, CAP's interventional measurement directly captures each head's functional contribution.'
The paper evaluates CAP on GSM8K, StrategyQA and ARC-Challenge using Llama-3-8B-Instruct and Mistral-7B-Instruct at 10%, 20% and 50% sparsity. At moderate sparsity, the method 'improves over Wanda in most model-benchmark configurations,' with the biggest reported gain on ARC-Challenge for Llama-3. Two researchers we follow flagged the preprint shortly after it went up.
The paper is upfront about where the method stops helping. The abstract reports that results remain 'limited by coarse MLP attribution at 50% sparsity.'
Shared on Bluesky by 2 AI experts
Originally reported by arxiv.org
Read the original article →Original headline: Pruning via Causal Attribution Preserves Reasoning Performance in Large Language Models