Abbeel, Fei-Fei team's robot exam caps coding agents at 34%

The strongest of eight coding agents tested on a new robotics benchmark, GPT-6 Astra in Codex, reaches a 34% success rate on 456 core tasks, according to a paper on arXiv from a team that includes Pieter Abbeel, Li Fei-Fei, Jiajun Wu, and Dawn Song.

The authors call it the Embodied First Exam: 1,156 tasks across 35 suites on 47 robots, including 9 new suites that reach past pick-and-place into spatial puzzles, dexterous manipulation, multi-robot interaction, dynamic whole-body control, and sim-to-sim engineering. The framework lets agents pick their own approach, "direct control, code to control, or train a policy to control."

Three evaluation settings. "The Standard Setting mimics the real world: the agent receives only sensor observations and, by default, cannot reset the environment," the authors write. A Realtime variant keeps the environment running while the agent reasons. The Privileged Setting hands over full simulator state and unlimited resets, serving as an approximate upper bound.

The 34% top score is on the Standard Setting. The paper frames it plainly, "leaving much room for improvement." To probe where agents trip, the authors label the stage at which failed attempts go wrong and compare failures across agents, settings, and categories.

The abstract publishes no per-suite breakdown, no second-place agent's score, and no numbers for the Privileged Setting the authors call an approximate upper bound.