SpectraReward turns pretrained MLLMs into image-gen rewards

The bottleneck in image-generation RL has not really been the RL. It has been everything upstream of it: collecting human preference pairs, training a reward model on those pairs, then hoping that reward model still agrees with people once the policy starts exploiting it. A new arXiv paper from Runhui Huang and colleagues, titled Read It Back, argues you can skip almost all of that.

Their method, SpectraReward, treats any pretrained multimodal LLM as a ready-made reward model. Instead of asking the MLLM to score an image or answer decomposed verification questions, they run a single image-conditioned, teacher-forced forward pass and use the average log-likelihood of the original prompt given the generated image as the reward. The intuition is that a good image should make the prompt easy to reconstruct. No preference labels, no reward-model fine-tuning, no separate training stage.

The more interesting variant is Self-SpectraReward. In unified multimodal models, where one stack does both understanding and generation, the policy's own understanding branch is the reward signal for its generation branch. That is a closed loop with no external supervision, and the authors report it matches or exceeds much larger external reward models, which they attribute to reward-policy alignment.

The scope of the study is where the claim gets its weight, or does not. They report running across two diffusion models, three RL algorithms, nine reward MLLM backbones from four MLLM families spanning 4B to 235B parameters, and five out-of-distribution text-to-image benchmarks. One of their headline observations is that larger reward MLLMs are not always better, which cuts against the default assumption that you throw the biggest available critic at the problem.

The honest caveat is that a log-likelihood reward measures reconstructability, not taste, and the abstract does not name the five benchmarks, so it is hard to tell how much of the win survives contact with production distributions or human aesthetic judgments the prompt does not fully pin down. What the paper does give you is a plausible way for small teams to bootstrap an RL loop on their diffusion models using MLLMs they already have on disk, and a hint that the unified-model architectures shipping this year may get a self-improvement mode almost for free.