Princeton paper: coding agents beat TAMP planners 95% vs 47%

Given only a task description and simulator access, Codex running GPT-6 Astra averaged 95% success on 16 robot task-and-motion environments, versus 47% for the hand-engineered planners that came with those benchmarks. Claude Code running Opus 5 averaged 82%. Codex running GPT-5.6 Sol averaged 56%. Those are the headline numbers in a new paper from Princeton, Carnegie Mellon and Cambridge evaluating off-the-shelf coding agents on generalized task-and-motion planning (TAMP).

The scale is unusual for TAMP work. The study covers 28 environments from the KinDER benchmark and three PDDLStream domains, five runs per method per environment, and 100 held-out test instances per synthesized program. "Across all program synthesis methods, we evaluate 980 generated programs on 100 held-out instances each, 98,000 evaluation episodes in total," the abstract states. Each synthesis run had a $20 model-usage budget; each evaluation instance had a 60-second wall-clock timeout.

The programs are fast at test time too. On the 14 environments with a planner and multiple object counts, Opus and Astra programs took 2.1 and 0.5 seconds per instance on average, compared with 29 seconds for the planners. Astra averaged 1.3 milliseconds per action, Opus 11.7 milliseconds. "As object counts grow, the agents' programs maintain higher success than the planner, using an order of magnitude less computation per instance on average," the authors write.

Traces from the runs make the physical-reasoning claim concrete. In the Shelf environment, one Opus run built a kinematic model of the Kinova Gen3 arm from its own knowledge without internet access, then used a grasped cube as a marker to calibrate joint offsets. Fitting six parameters cut the RMSE between predicted and observed cube positions from 38.9 to 1.8 mm. In one Sol run on the Blocked task, a commit adding a fallback to a distant table lifted success from 15% to 56%. The paper says the agents also found "non-prehensile maneuvers and uses of the environment layout that, to our knowledge, have not appeared in prior work on these benchmarks."

Failures remain. Some Dynamic3D environments, those requiring sweeping or pouring many small objects, are still largely unsolved in the main setting. Giving Claude Code the environment source code raises mean success from 74% to 84%; giving it to Astra raises it from 86% to 95%. The non-agentic LLMGenPlan baseline, run on Opus 5 with the same $20 budget and the same source code, reaches 28%.

The result adds to a run of coding-agent-plus-robotics work we've been tracking; last week's EmbodiedSWE used frontier coding agents to generate robot training data. All code and prompts are released.