RPO claims nearly 4x faster diffusion RL via attention routing

A new preprint on arxiv argues that reinforcement-learning alignment of image and video generators can be sped up by close to four times, not by cutting rollouts but by moving where the randomness lives: into the model's attention routing.

The method, Routing Policy Optimization, inverts the usual score-function setup for diffusion RL. Existing methods, the authors write, inject exploration by "perturbing the reverse denoising trajectory, leading to costly rollout-based optimization." RPO instead works "by turning attention routing into a stochastic exploration action." At selected denoising levels, each query block keeps its own block and samples further key blocks in proportion to the generator's native attention mass, or keeps dense attention; a Gaussian likelihood around the dense velocity scores the perturbation.

The paper also describes "a controlled two-rollout estimator": two rollouts share the same prompt, initial latent, routed denoising levels, and dense anchor layers, differing only in their sampled attention routes, with the reward gap supplying the advantage signal.

The headline claim: experiments on text-to-image and text-to-video generation show RPO "improves generation quality while achieving nearly 4x higher training efficiency than Flow-GRPO-style methods." At inference, the authors write, "RPO restores dense attention and retains the original deterministic sampler," so the model that gets served is the one the pretraining pipeline already produced.

The abstract names no specific base models and publishes no per-benchmark numbers behind the nearly-4x figure. It remains a single-group preprint, uncorroborated by outside replication.