Elastic Pruning Cuts DINOv3 ViT-H 50% at 0.8% Accuracy Loss

A single pre-trained vision transformer can reportedly be cut in half at inference with almost no accuracy damage and no retraining. The claim comes from a paper posted on Hugging Face and titled "Your Transformer Is Secretly Elastic." The authors report that "pruning DINOv3 ViT-H+/16 to 50% sparsity removes 427M parameters while reducing ImageNet-1k linear-probing accuracy by only 0.8%, without the need for weight correction or retraining."

The mechanism is a differentiable soft mask placed over every MLP neuron and attention head, with the pre-trained weights frozen. The paper describes its contribution as "a differentiable relaxation of the functional unit-ranking problem," and says that "unlike prior work, which optimizes a single importance weight per MLP block or attention head, we rank every individual MLP neuron and attention head jointly, on a common scale, by minimizing the model's loss under soft pruning across many sparsity targets, all at once."

The sweep is wider than a typical pruning paper. The authors test "27 vision and text transformer encoders ranging from 22M to 8B parameters," naming DINOv3, DeiT, SigLIPv2, BGE-L, E5-L and Qwen3 among the targets, and report smooth degradation up to 60% sparsity.

The deployment hook is throughput. "At 60% sparsity, a pruned AugReg ViT-B/16 model achieves up to 1.80× higher throughput and 1.26× lower single-query latency on an A100 GPU," the paper states, with switching between budgets costing 0.46 ms of mean overhead and the elastic model "produced in minutes, with construction time scaling sub-linearly with parameter count."

Not everything scales symmetrically. The paper's conclusion flags that decoder-based text embedders, the paper's own example being Qwen3, hold up less well at deep sparsity than vision or bidirectional text encoders. No per-task MTEB figures for the 8B Qwen3 setting appear in the abstract. The paper lands alongside last week's FreeMatching fusion of FLUX.2 and DINOv3 in our computer vision tracker.