Debias-SparseGPT Adds Fairness Term to LLM Weight Pruning

Weight-sparsification pruning like SparseGPT can amplify a language model's biases, and a new method claims to fix that without losing accuracy or perplexity. In a paper posted on Hugging Face, Irina Proskurina, Guillaume Metzler, Antoine Gourru, and Julien Velcin introduce Debias-SparseGPT, a post-training pruning method that adds 'a second-order term defined over demographically contrasting inputs' to the standard SparseGPT objective.

The failure mode the authors name is specific: 'weight sparsification methods, such as SparseGPT, can amplify existing biases in models, with outputs varying significantly depending on persona cues in the prompt.'

Across three sparsity regimes (25%, 50%, and structured 2:4), the method 'consistently reduces pruning-induced bias compared to SparseGPT while preserving model perplexity and zero-shot accuracy,' the abstract reports. Under the most restrictive 2:4 pattern, which most aggressively degrades model quality, adding 'long-context, content-rich examples' to the calibration set further lifts both fairness and downstream performance, the authors write.

Per-model bias deltas and named benchmark scores are not in the abstract. The paper lands inside a heavy quarter for open-source AI work on our tracker, alongside 90 AI ethics stories over the same 90 days.