paper web signal

SimAudit finds 61 defects in robot manipulation benchmark checkers

TL;DR

  • SimAudit, an LLM-based pipeline, flagged 61 distinct defects across 49 tasks in RoboTwin 2.0, RoboCasa365, and RoboDojo success checkers.
  • The audit catalogued 40 phantom successes (checkers accepting partial work) and 21 silent failures (checkers rejecting valid completions) from 183 test cases.
  • Error patterns differ: RoboCasa365 mostly over-accepts, RoboDojo mostly over-rejects, and both error types were common in RoboTwin 2.0.

Success checkers used across three simulated robot manipulation benchmarks mislabel policy attempts often enough that an audit posted to arXiv, titled "Phantom Success, Silent Failure," found 61 distinct defects where the automated grader either accepted a partial attempt or rejected a valid completion. The paper introduces SimAudit, an LLM-based pipeline that compares a task's natural-language instruction against its checker code to identify likely disagreements, then recreates those scenarios in simulation and verifies behavior with multi-view renderings.

Across RoboTwin 2.0, RoboCasa365, and RoboDojo, 183 simulated test cases "produced the predicted checker verdict," and after visual verification and deduplication the authors report "61 distinct defects across 49 tasks, including 40 phantom successes and 21 silent failures." The pattern is not uniform by benchmark. "Most defects in RoboCasa365 accepted incomplete executions, most in RoboDojo rejected valid executions, and both error types were common in RoboTwin 2.0," the abstract states.

The authors frame the implication as a protocol problem rather than a benchmark-specific bug, arguing that validating success checkers "requires testing both incomplete executions and valid alternatives to expert demonstrations." The abstract publishes no per-task defect list and does not quantify how patching the checkers would reorder any existing leaderboard.