huggingface.co web signal

TCP4D Unifies Robot Tool-Point Tracking and Trajectory Generation

TL;DR

  • The tracker reports an absolute trajectory error of 7.60 mm on DROID and 16.07 mm on the in-house RoboLab dataset.
  • RoboTwin success is 69.19% under the clean2clean setting but drops to 10.94% under clean2random in the same tables.
  • Joint training on augmented cross-embodiment data lifts ARX-X5 success from 30.5% to 72.0%, a 41.5-point gain over augmented-only.

TCP4D, a new model from HKUST, Beijing Institute of Technology and collaborators at NUS, PolyU, Sun Yat-sen and Zhejiang, treats a tool-center-point pose as the action space for a robot and learns both to track that point from monocular RGB video and to generate future trajectories from a language instruction, inside a single 4D scene representation. On the DROID dataset the tracker reports an absolute trajectory error of 7.60 mm; on the in-house RoboLab set that figure rises to 16.07 mm. The paper page on Hugging Face states 69.19% mean success on RoboTwin and a 78.33% average success rate across two real-world bimanual tasks, Placing Pens and Stacking Bowls.

The headline numbers sit next to a visible caveat in the paper's own tables. The 69.19% RoboTwin figure is the clean2clean setting; under clean2random the same model falls to 10.94%, which puts a hard ceiling on how far current performance carries outside matched conditions. The real-world average comes from two tasks on a single CoBoT Magic bimanual platform, not a broad suite, and the ten-shot adaptation experiment is reported on a UR5 arm with an 8.85 mm ATE.

The more consequential claim is the cross-embodiment augmentation pipeline, which converts existing RGB demonstrations into training data for a different arm by retargeting the recovered TCP motion and synthesizing fresh observations, with no recorded source-robot states required. The authors write that "joint training on source and augmented demonstrations improves target-robot success by 41.5% over augmented-only training," with ARX-X5 moving from 30.5% to 72.0% in their tables, and task-level jumps like Click Bell going from 14% to 70%. It is the latest entry in a busy 90-day stretch of robotics work we're tracking, much of it aimed at the same per-robot data budget.