TCP4D Unifies Robot Tool-Point Tracking and Trajectory Generation
TL;DR
- The tracker reports an absolute trajectory error of 7.60 mm on DROID and 16.07 mm on the in-house RoboLab dataset.
- RoboTwin success is 69.19% under the clean2clean setting but drops to 10.94% under clean2random in the same tables.
- Joint training on augmented cross-embodiment data lifts ARX-X5 success from 30.5% to 72.0%, a 41.5-point gain over augmented-only.
TCP4D, a new model from HKUST, Beijing Institute of Technology and collaborators at NUS, PolyU, Sun Yat-sen and Zhejiang, treats a tool-center-point pose as the action space for a robot and learns both to track that point from monocular RGB video and to generate future trajectories from a language instruction, inside a single 4D scene representation. On the DROID dataset the tracker reports an absolute trajectory error of 7.60 mm; on the in-house RoboLab set that figure rises to 16.07 mm. The paper page on Hugging Face states 69.19% mean success on RoboTwin and a 78.33% average success rate across two real-world bimanual tasks, Placing Pens and Stacking Bowls.
The headline numbers sit next to a visible caveat in the paper's own tables. The 69.19% RoboTwin figure is the clean2clean setting; under clean2random the same model falls to 10.94%, which puts a hard ceiling on how far current performance carries outside matched conditions. The real-world average comes from two tasks on a single CoBoT Magic bimanual platform, not a broad suite, and the ten-shot adaptation experiment is reported on a UR5 arm with an 8.85 mm ATE.
The more consequential claim is the cross-embodiment augmentation pipeline, which converts existing RGB demonstrations into training data for a different arm by retargeting the recovered TCP motion and synthesizing fresh observations, with no recorded source-robot states required. The authors write that "joint training on source and augmented demonstrations improves target-robot success by 41.5% over augmented-only training," with ARX-X5 moving from 30.5% to 72.0% in their tables, and task-level jumps like Click Bell going from 14% to 70%. It is the latest entry in a busy 90-day stretch of robotics work we're tracking, much of it aimed at the same per-robot data budget.
Originally reported by huggingface.co
Read the original article →Original headline: TCP4D Unifies Tool-Center-Point Tracking and Trajectory Generation for Robots, Hits 78% Real-World Success From Video