OSReward: frontier VLM judges overrate agent success

The bit worth pausing on in this week's arxiv drop is not another agent benchmark; it is a benchmark for the judges of agent benchmarks. A team led by Qiushi Sun has released OSReward, a standardized evaluation for the vision-language models that are increasingly used to decide whether a computer-using agent actually completed a task. The finding is uncomfortable for anyone trusting those numbers: every state-of-the-art VLM the authors tested shares what they call a systematic leniency bias, marking failed trajectories as successes.

That matters more than the usual leaderboard shuffle because the VLM-as-judge is load-bearing infrastructure now. It scores public CUA benchmarks, and it supplies the reward signal for the RLAIF loops that many teams use to train the next generation of agents. If the judge is consistently generous, benchmark scores drift upward on their own and agents get trained to produce trajectories that look successful to a lenient rater rather than trajectories that finish the job.

The authors' proposed fix is a pair of open reward models, OS-Shepherd-9B and OS-Shepherd-35B, that they say match commercial judges at 30 to 60 times lower cost. They have released the models, the OSReward benchmark, and an annotated 100K trajectory dataset called OS-Shepherd-100K. The pitch is not that these are better judges than a frontier VLM in absolute terms, but that they close enough of the gap to make cheap, continuous auditing of an agent stack finally practical for teams that were priced out.

The honest caveat is that the leniency-bias claim and the cost-parity numbers both come from a single team's evaluation, and the paper as summarized here does not spell out which specific frontier VLMs were tested or how OS-Shepherd behaves on trajectories outside its training distribution. Take the specifics as reported, not settled. Even so, if you are shipping or scoring a computer-using agent right now, the useful move this week is to re-run a sample of your evaluations through an independent judge and see how much of your reported success rate survives.