SO-101 Bench Shows Sim Policy Rankings Match Real Hardware

Five robot policies trained without any real data from the test rig ranked identically on the real SO-101 arm and in its digital twin, with Spearman ρ=1.00 and a mean maximum rank violation of zero. That is the sim-to-real correspondence number from SO-101 Bench, a new Isaac Lab benchmark built around a hobbyist arm that costs $120–150 in parts, against roughly $4,900 for an ALOHA setup.

The benchmark runs 916 feasibility-checked episodes across four tasks and three generalization splits (seen, unseen object of a seen class, unseen class) over 138 household objects in simulation. The authors also release around 4,400 real-robot teleoperated episodes totalling about 20 hours.

What makes the leaderboard awkward for the usual VLA story: the top entry is not a vision-language-action model at all. 'GPT-6.1 Sol with PyRUA-Lean harness achieves 83.2% vs. 60.8% for best learned policy,' the paper reports, with the LLM-agent stack delegating grasps to a Galaxea 0.5 submodule. On a 60-episode harness ablation, the same base model drops from 85% with that harness to 35% when it issues direct tool calls, and sits at 53% when Galaxea 0.5 runs on its own. The paper's blunt summary: 'harness matters more than model for LLM agents.'

The caveats are concrete. Across the five-policy study, simulated success overestimates real success by 9 to 31 percentage points depending on the policy. GPT-6.1 Sol loses 20 points on hardware; GR00T N1.7 and π₀.₅ each lose 31. The sim2real study spans 45 layouts in one environment, and the confidence interval on the MMRV ranking metric runs from 0.000 to 0.169. The simulator also pauses during LLM inference, hiding the agent's wall-clock cost: 197 seconds median against 38 for the Galaxea VLA.

The SO-101 is already the default low-cost arm in the LeRobot community: 559 of 976 LeRobot-tagged datasets use the SO-100 or SO-101, and SmolVLA was pretrained on 481 community datasets. AI Weekly has logged 169 robotics stories in the last 90 days; this one landed the same day as the Rollback World Model result that beat RISE 24%.