Self-OPD Drops Teacher Model From Flow Matching Distillation

Self-OPD replaces the pre-trained specialized teacher in on-policy distillation with the flow matching student's own self-rollouts, according to a preprint posted to arXiv by Shiyi Zhang, Mushui Liu and nine co-authors.

The authors argue the standard OPD recipe, which has "achieved significant success in Large Language Models (LLMs) and has recently been adapted to flow matching models," carries "two major issues": training "a separate, task-specific teacher for every new objective incurs high computational costs," and teacher-student distribution mismatch "often leads to compounding errors along the generation trajectory."

At each timestep the student branches its deterministic next-state prediction into "K stochastic SDE candidates," rolls each out with an ODE sampler, and scores them against a deterministic self-reference baseline to obtain normalized advantages. High-advantage branches pull the velocity field toward them and low-advantage ones repel it, under what the paper calls an "all-branch pull-push objective" with "direction-aware attenuation and SDE-variance normalization." For multi-objective alignment, Self-OPD "fuses normalized scores at the reward level, avoiding direct gradient conflict."

On single and mixed reward benchmarks, the authors report that Self-OPD "outperforms prior RL and OPD methods without task-specific teachers." The abstract does not name the baselines, list the models, or publish per-benchmark numbers.