ShortOPD Recovers Pruned LLMs 9× via Suffix-Repetition Fix

Structured pruning cuts an LLM down to a smaller size and, on multiple-choice benchmarks, the compressed model looks fine. But when you actually deploy it to generate free-form text, it falls over. That is the observation behind ShortOPD, a new paper on arXiv, and it is worth reading if you have ever been surprised by a distilled or pruned model that scored well in eval but produced junk in production.

The authors' framing is that useful generations are demoted, not erased after compression. In their terms, pass@k recovers substantially even when pass@1 collapses, which suggests the knowledge is still in the weights but the top-of-distribution ordering is broken. When you push the recoverable regime, what breaks it is suffix repetition. The model spirals into repeating tokens, and the training budget gets spent on that low-information tail instead of actual recovery.

ShortOPD's fix is a scheduling strategy that detects teacher-confirmed repetitive suffixes and allocates training based on effective sequence length rather than the full rollout. The reported numbers are striking. Recovery is raised to roughly 9× the unrecovered baseline, with 1.6 to 4.4× gains over standard recovery methods like SFT, KD and SeqKD. It reportedly matches an 8192-token baseline in 25% of the wall-clock time (8.5 hours vs 35.9), and cuts token consumption by 71% versus fixed-horizon rollouts. Evaluation spans mathematics, code generation, and open-ended text.

The honest caveat is that these numbers are from the authors' own runs, and the paper as summarized does not tell you which base model sizes were tested, or how the method interacts with quantization, which most deployment stacks combine with pruning. Independent replication is going to matter before anyone reprices a compression pipeline around it.

If the result holds up, the real winners are teams shipping pruned or distilled models to production where free-form generation is the actual product, not the benchmark score. Cheaper recovery at a quarter the training time changes what is economically viable to compress and ship, especially for on-device or budget-constrained deployments.