Zhejiang-Baidu Paper: Weak Decision Actors Still Teach Strong RL

A decision model that completes only 42.9% of ALFWorld tasks when it chooses the actions itself can lift a reinforcement-learned policy from 72.8% to 82.9% success on the same tasks, if it is asked the right question after the episode ends.

That is the central finding of "Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon Agents", posted November 13 by a group from Zhejiang University and Baidu. The team compares three decision models (JEV, Decider v1.2, and GPT Decisions) against three general LLMs set up to answer the same questions, on ALFWorld and WebShop.

As actors, all six models get pulled off the required step by an easier one. The paper reports: "JEV, for example, completes 70.8% of tasks with two independent subgoals but only 14.7% of tasks with ordered ones." The models are not missing the knowledge. Remove the easier subgoal from the goal description and JEV's probability on the required state change rises from 14.6% to 77.6%; a plain reminder about the dependency only shifts success by -2.8% to +5.2%.

The turn comes when the same models look at a finished trajectory. Asked whether each action helped, rather than whether the task will succeed, JEV scores correct steps 0.36 higher on average within a rollout, versus a gap of 0.05 for the outcome question. Plugging those per-step scores into GRPO as a teacher, JEV raises a Qwen2.5-1.5B policy from 72.8% to 82.9% on ALFWorld and from 56.8% to 71.4% on WebShop. Asking the Value question on ALFWorld instead yields 69.3%. The authors write that "the 1.5B policy taught this way outperforms a 7B policy trained with outcome rewards alone on both environments".

The cost claim is specific: "step-level supervision can be obtained cheaply by querying a frozen decision model after each episode, without step labels or a learned critic, for 22 to 30 USD of queries per training run in our experiments." Code is released at ZJU-REAL/decision-agents. That $22-to-$30 figure is not broken out across JEV, Decider v1.2, and GPT Decisions, and the training-side numbers come from a single 1.5B base.

The result sits alongside a steady run of fine-tuning work we have tracked that routes a frozen signal back into agent RL.