Arxiv Paper Says LLM RL Optimizes the Wrong Policy, Offers MIPI

A new arxiv paper from a team led by Jing Liang is worth reading if you spend any time thinking about the last mile of large language model RL. Its core claim is quiet but load-bearing. When a lab trains a reasoning model with reinforcement learning, the policy that gets optimized on the training engine is not the same as the policy that actually runs on the inference engine, even after parameters are synchronized. The two engines assign different probabilities to the same trajectories, and that gap gets pushed under the rug in most reported results.

If you take the argument at face value, it reframes what a well-behaved training curve is actually telling you. The authors call this an objective misalignment. Improving the training policy does not automatically improve the inference policy that a user samples from, so the loss you are staring at during a run is a proxy for something a step removed from what ships. Their fix, which they call MIPI, or Monotonic Inference Policy Improvement, is an objective that targets the inference-side policy directly. It is applied through a two-step update framework named MIPU that generates candidate updates and selectively accepts them using an inference-side gap proxy.

The paper reports that on two model scales, under conditions with a high training-inference mismatch, MIPU improved average reasoning performance and training stability against baselines. That points in a direction many people have quietly suspected, but take the specifics as reported and not settled. The abstract does not name the benchmarks, does not give point estimates, and does not compare against the specific PPO or GRPO variants labs are running in production today, so it is hard to say whether the effect is a small tidy correction or a large one.

The honest caveat is that a lot of the value here depends on how the idea survives contact with the open source RL frameworks, since that is where a mismatch fix would either compound across teams or quietly wash out. What the write-up does not give you is any read on cost, so it is unclear whether the inference-side gap proxy is cheap enough that a small team can adopt it, or expensive enough that only well-funded labs will bother.

If the effect is real and cheap to apply, the beneficiaries are the teams doing reasoning RL on modest compute, because a stability win that closes the gap between training loss and deployed behavior is worth more to them than to the labs already brute-forcing around it.