Found first: a primary source the press has not covered yet.
An LLM-based pipeline called SimAudit found 61 distinct defects across 49 tasks in three widely used robot manipulation benchmarks, according to a new paper by Zirui Song, Jiadong Zhang, Zheng Lu, Xiaosong Ma, Xiaojun Chang, and Xiuying Chen. The defects split into 40 phantom successes, where a benchmark's success checker accepts incomplete task execution, and 21 silent failures, where valid completions are rejected.
What the source says
SimAudit works by analyzing checker code against task instructions, generating disagreement scenarios, recreating them in simulation, and using multi-view renderings for visual verification. The pipeline produced 183 simulated test cases; after visual verification and deduplication, 61 distinct defects remained across 49 tasks. RoboCasa365 concentrated its defects in phantom successes, with checkers accepting incomplete executions. RoboDojo skewed toward silent failures, rejecting valid completions. RoboTwin 2.0 showed both types. Author affiliations are not listed in the preprint.
Why it matters
Phantom successes inflate policy scores; silent failures suppress them. Any published result against these three benchmarks carries that uncertainty unless the affected tasks have been manually verified. The paper's core argument is that checker validation requires testing against incomplete executions and valid alternatives beyond expert demonstrations. Existing leaderboard comparisons on RoboCasa365, RoboDojo, and RoboTwin 2.0 do not carry that guarantee.