Paper: Fixed Steering Vectors Match 90%+ of RL Gains on SD3.5, FLUX
TL;DR
- Fixed per-block steering vectors added to a frozen diffusion transformer recover 90.0–96.3% of RL reward gains across six image models.
- The full vector set uses 288,768 parameters on SD3.5-M, about 1.5% of a rank-32 LoRA's 18.8M, and transfers across resolutions.
- Trained from scratch with Flow-GRPO, steering vectors alone reach 50–52% win rate against the RL-tuned models in a user study.
288,768 parameters, added as a constant offset to a frozen Stable Diffusion 3.5 Medium, reproduce roughly 90% of what Flow-GRPO reinforcement learning gets from updating 18.8 million. That is the top-line claim of a new paper on Hugging Face from researchers at fal, Georgia Tech, IBM Research and Virginia Tech, titled "RL Post-Training in Diffusion Transformers Is Activation Steering."
The authors fit per-block, input-independent steering vectors to match the velocity output of an RL-tuned model, without touching the base weights. Across six RL-tuned image diffusion models and two video models, the paper reports "recover 90.0–96.3% of reward gains on six RL-tuned image models and 89.8–111.9% on two RL-tuned video models," using about 1.5% of a rank-32 LoRA's parameters.
The vectors also hold up when trained from scratch. Running Flow-GRPO directly on the steering vectors, starting from the frozen base, the authors report 103.6% of the original RL PickScore gain and 104.6% of the OCR gain, with a user study putting the steered output at a 50–52% win rate against the fully RL-tuned models, i.e. statistical parity.
One passage states the takeaway bluntly: "most of these improvements can be reproduced through fixed steering vectors added to block outputs of a frozen base model, without updating model weights." The authors argue joint attention converts those constant shifts into prompt-dependent outputs; the base generator is already doing the prompt-specific work, and RL mostly nudges it toward outputs it already knows.
A limit sits in the appendix. On supervised fine-tuning tasks like DreamBooth, the same technique recovers only 68–95% of subject similarity, trailing a same-size LoRA at 81–100%. Fitting the full vector set takes up to 17 H100-GPU-hours per image model, up to 70 for video.
It lands in a run of this week's cheap-post-training results, after CurvDelta's low-curvature weight-update confinement a day earlier in the fine-tuning feed.
Originally reported by huggingface.co
Read the original article →Original headline: Paper: 90%+ of RL Post-Training Gains in Diffusion Transformers Come From Fixed Steering Vectors