RoboColosseum: policy rankings don't transfer across robots

Across 5,699 pairwise comparisons at 124 sites, which robot policy 'wins' depends mostly on which body you put it on. That is the finding of RoboColosseum, a distributed evaluation of five vision-language-action policies and three frontier AI agents across four embodiments: SO-101, Franka, Bimanual-YAM, and the G1 Humanoid.

The paper reports that after each policy received the same 100 demonstrations per task on 20 tasks, "GR00T N1.7 leads on SO-101 and pi0.5 leads on Franka and Bimanual-YAM." Across the embodiment-specific task suites, the authors put the ranking agreement at Kendall W = 0.24, and write that Open Track rankings "poorly predict relative performance after fine-tuning, with substantial rank reversals on all three embodiments shared between tracks."

The humanoid result is the blunt one. "On the G1 Humanoid, no policy completes any of the five tasks, yet average progress ranges from 13.3% to 23.2% across policies." The authors argue this is exactly why graded scoring matters: "measuring partial progress reveals differences that task success alone misses."

Two other claims sit in the abstract without numbers attached. Frontier agents, used without task-specific adaptation, "rival specialized VLA policies" in the Open Track; and active matchmaking, the paper says, "achieves comparable ranking agreement with substantially fewer physical trials than uniform sampling." The abstract does not break out per-agent scores, name the three frontier agents, or list the 20 tasks.