Negative Self-Distillation Trains LLMs by Avoiding Flaws
TL;DR
- A new arXiv paper proposes Negative Self-Distillation (NSD), which trains LLMs to reason by pushing them away from flawed traces instead of imitating good ones.
- The authors argue standard On-Policy Self-Distillation can harm LLM performance on complex reasoning tasks by suppressing uncertainty and exploratory behavior.
- NSD adds a dynamic gating mechanism that isolates reasoning-critical tokens, aimed at keeping foundational language skills intact during training.
A paper posted to arXiv on September 10 proposes flipping how language models learn to reason: instead of imitating good reasoning, make them run away from bad. The preprint, from Rongcan Pei, Zhepei Wei, Shuyao Xu, Xinyu Zhu, Wei-Lin Chen and Yu Meng, calls the technique Negative Self-Distillation, or NSD.
It is set up as a direct rebuke of the standard alternative, On-Policy Self-Distillation. NSD generates its own negative examples, including a "careless reasoner" persona, and pushes the model to "diverge from flawed reasoning rather than imitating privileged solutions."
A second piece of the method is a "dynamic gating mechanism that automatically identifies and isolates reasoning-critical tokens," meant to keep the model's foundational language skills intact while it learns to avoid flawed patterns.
The 23-page paper reports that "NSD consistently outperforms OPSD and other label-free, self-bootstrapping reinforcement learning baselines." The abstract publishes no per-benchmark numbers, and does not name the base models tested. Two researchers we track shared the preprint the day it went up.
Shared on Bluesky by 2 AI experts
Originally reported by arxiv.org
Read the original article →Original headline: Negative Self-Distillation: Learning to Reason by Avoiding Flaws