P²-MOPD hits 94.7% distillation recovery in half the steps
TL;DR
- A 1.7B student trained with P²-MOPD reaches 94.7% average capability recovery across four domains, versus 80.9% for standard multi-teacher on-policy distillation.
- The method matches MOPD's final average recovery in half the training steps, tested at 1.7B and 4B scales from the same initialization.
- P²-MOPD jointly optimizes mixing coefficients and student weights under one on-policy distillation objective, keeping the student trainable rather than frozen.
A 1.7 billion-parameter student model trained with P²-MOPD reaches 94.7% average capability recovery across four domains, versus 80.9% for standard multi-teacher on-policy distillation, according to an arXiv preprint from Sinuo Wang, Jiacheng Lu and co-authors.
The method treats the linear combination of student and expert parameters as one differentiable model and, in the authors' words, "jointly optimizes the mixing coefficients and the student weights under the same on-policy distillation objective." Unlike task-arithmetic merging, which freezes the components, P²-MOPD keeps the student trainable, so the experts "act as parameter priors without confining it to this span." Tested at 1.7B and 4B scales from the same initialization, it matches MOPD's final average recovery "in half the training steps."
The abstract reports only the aggregate figure across the four domains and does not name them, leaving per-domain recovery and any regressions unclear.
Shared on Bluesky by 1 AI expert
Originally reported by paper
Read the original article →Original headline: P²-MOPD Clears Multi-Teacher Distillation Ceiling in Half the Training Steps