RoboDojo unifies sim and real robot manipulation benchmarks

Robot manipulation research has a persistent comparability problem. Sim results and real-world results usually come from different setups, in different labs, on different hardware, and rarely line up against each other in a way you can trust. A new arxiv paper called RoboDojo tries to close that gap with a single benchmark that covers both sides at once.

The paper, led by Tianxing Chen with a large team of co-authors, describes a benchmark that bundles 42 simulation tasks and 18 real-world tasks into one evaluation layer. The simulation side runs on Isaac Sim and grades policies on five dimensions: generalization, memory, precision, long-horizon execution, and open-vocabulary instruction following. The real-world side, called RoboDojo-RealEval, is what makes this different from most previous efforts, offering a reproducible real-world evaluation system with remote cloud access, standardized hardware, scene reset, evaluation protocol, and a deployment interface, so a lab that does not own the robot in question can still run its policy on the real thing.

The authors integrated 30 policies into a wrapper called XPolicyLab and evaluated all of them across both benchmarks, publishing a public leaderboard alongside the paper. The pitch to policy developers is that you wire your policy in once and it gets scored across the whole sim and real stack with minimal adaptation.

The honest caveat is that this is a benchmark introduction, and benchmark papers tend to look better on release than in day-to-day use. The abstract does not tell you which specific robot arm the real-world layer standardizes on, where the hardware physically lives, how external teams queue for access, or how tightly sim scores actually track real scores for the 30 policies already run. Those are the things that determine whether this becomes the shared measuring stick the field has been missing or another leaderboard people mostly ignore.

If it works, the direct beneficiaries are academic labs without expensive manipulator hardware and foundation model teams that want a common yardstick, and the second-order effect is that "we beat X on the benchmark" claims start meaning something more concrete.