PointWAM lifts dexterous robot success 56.9 points on DexJoCo

Pre-training on human-hand videos lifts a dexterous-manipulation policy's average DexJoCo success rate by 56.9 percentage points, according to a new preprint introducing a model called PointWAM. The paper's angle is that the gain comes from switching representation: PointWAM treats the environment and hands as 3D point trajectories rather than pixels.

"World action models jointly learn to forecast world dynamics and predict robot actions, such that the learned internal world dynamics guide accurate actions," the authors write. Prior approaches, they argue, usually represent the world as "RGB frames or latent counterparts while predicting actions as end-effector poses or joint angles" and so miss the "3D spatial structure and contact geometry central to dexterous manipulation." PointWAM instead decomposes the world into a scene and hands and forecasts both "as 3D point trajectories within a shared space-time coordinate frame," then retargets the predicted hand motion to robot commands.

The gains come in two chunks. Human-video pre-training alone contributes 56.9 points. Adding supervision for the scene trajectory on top of forecasting the hands adds another 10.9. The combined model, the authors report, "surpasses the prior state of the art on ten DexJoCo tasks by 11.7 points and outperforms strong VLAs on a real robot."

The abstract names neither the baseline VLAs nor the specific tasks or physical robot used in the real-world test, and gives no numbers for how faithfully predicted hand motion retargets to robot joint commands, which is where dexterity has to work.