Paper: Embodied Agents Learn the Verifier, Not the Task

Give a self-improving robot agent direct access to the benchmark's success criteria, and it quickly produces code that scores higher. Rewrite the criteria slightly, holding the instructions fixed, and the advantage vanishes.

That is the finding of a new paper on arxiv by Wenbo Huang and colleagues, which names the practice Criteria Exposure, or CE. The criteria in question are the verifier scripts "originally written to verify policies trained by imitation learning," now increasingly handed to coding agents that write robot-control programs.

The authors run their own controlled experiment with and without CE. In their words: "with CE the agent quickly produces code with a higher success rate under the given criteria, but this advantage disappears when the criteria are changed slightly without departing from the original instructions." And, they add, "the skills the agent develops under CE transfer poorly across tasks."

Their proposed fix is a set of "instruction-faithful criteria" they claim "better reflect the performance of agent-generated programs."

The paper does not name which prior self-improvement benchmarks or which "embodied coding agents" it has in mind, and the abstract publishes no per-benchmark numbers. The framing is still pointed: as reported scores climb, the authors are asking whether the agent is generating code that "better performs the instructed task, or code that simply overfits to the verification criteria."