Paper: Q-function pretraining barely helps online RL fine-tuning

A counterintuitive result from an arXiv paper this month caught my eye because it pokes at one of the assumed steps in the value-based RL fine-tuning recipe. The claim, from a paper by Perry Dong, Ron Polonsky, Dorsa Sadigh and Chelsea Finn on arXiv, is that naive Q-function pretraining on offline data often provides little benefit over random initialization when you go to fine-tune a pretrained policy online.

The reason they give is a specific mismatch. The Q-function you learn during pretraining is targeting the pretrained policy's Q-function, not the Q-function that online fine-tuning actually converges to, and that gap persists even after offline value maximization. So the effort you spend pretraining a Q-function is aimed at the wrong target, and more offline training does not close the gap.

Their proposed fix, called Initialization via Policy Ensemble or IPE, is straightforward. Train several diverse policies, pool their rollouts, and use that pooled data to bootstrap Q-function learning once online RL starts. Across a suite of challenging continuous control benchmarks the authors report an average 1.26x improvement in fine-tuning performance over naive Q-function pretraining.

The honest caveat is what the abstract does not spell out. 1.26x is an average across unnamed continuous control benchmarks, so the per-environment variance is not visible from the summary. It also does not quantify the compute cost of training an ensemble of diverse policies instead of a single one, and it does not test whether the finding transfers to discrete action spaces or to the LLM RL fine-tuning setting where a lot of current attention sits.

If it holds up, the practical upside is small but real. Teams doing robotics and continuous control fine-tuning may get to skip a pipeline step that was assumed to help, and replace it with something simpler and modestly better on average.