Deletion Tuning matches fine-tuning by zeroing 0.05% of weights
TL;DR
- Zeroing fewer than 0.05% of a pretrained LLM's weights matches full fine-tuning across RL with verifiable rewards, SFT, and on-policy distillation.
- The method, called Deletion Tuning, learns a binary mask over gate projections and ships hundreds to thousands of times smaller than an FFT checkpoint.
- Under reinforcement learning, Deletion Tuning's prior-task accuracy stays near the base while FFT, LoRA and random sparse fine-tuning lose up to 5.4 points.
"Zeroing fewer than 0.05% of its weights, with no other modification, matches full fine-tuning (FFT) at scale across three LLM post-training regimes," according to a NeurIPS 2026 paper from Jack Lu, Zhenbang Yang, Mike Lasby, Tejas Pote, Yani Ioannou and Mengye Ren.
The authors call the technique Deletion Tuning. It learns a binary mask over the gate projection of transformer blocks. No weight is updated; some are simply turned off. The resulting mask, the paper reports, is "hundreds to thousands of times smaller than the corresponding FFT checkpoint."
The three regimes tested are reinforcement learning with verifiable rewards, supervised fine-tuning, and on-policy distillation. Deletion Tuning also appears to blunt catastrophic forgetting: under reinforcement learning, its "prior-task accuracy stays near the base level while FFT, LoRA, and random sparse fine-tuning lose up to 5.4 points."
To ask why masking alone suffices, the authors examine the geometry of the learned weight changes using stable rank, spectrum drift and principal-weight overlap. "Compared with LoRA and random sparse fine-tuning, Deletion Tuning is closest to FFT across the three measures," they write. No model sizes or per-benchmark accuracies appear in the abstract.
Originally reported by paper
Read the original article →Original headline: NeurIPS 2026: Zeroing 0.05% of LLM Weights Matches Full Fine-Tuning Across All Three Post-Training Regimes