SJTU-Noematrix SWAM Lifts Real-World Robot Success to 40%
TL;DR
- SWAM is trained on a pool of ~47,000 hours of general video, egocentric footage, UMI demonstrations and robot demonstrations.
- Post-trained on RoboTwin Clean only, SWAM reports 88.10% success on Clean-to-Clean and 61.56% on Clean-to-Random in simulation.
- On three real-world tasks SWAM reports 40.00% success versus 23.33% for the strongest baseline; data, code and weights are released.
The abstract claims 40.00% success on three real-world tasks versus 23.33% for the strongest baseline. The model behind the number is SWAM, from a Shanghai Jiao Tong University, Shanghai Innovation Institute and Noematrix team studying how language-video alignment survives when action learning is added to a world-action model.
They curated "approximately 47,000 hours of general videos, egocentric videos, UMI demonstrations, and robot demonstrations," per the arxiv paper, then ran ablations over which supervision signals feed the model.
The practical finding, in the authors' own words: "Under a fixed training budget, mixing egocentric videos and UMI demonstrations improves control generalization. Model architecture affects how effectively these experiences are used, while scaling pretraining under the selected configuration brings further gains."
Post-trained on RoboTwin Clean demonstrations only, SWAM reports 88.10% on the Easy (Clean-to-Clean) and 61.56% on the Hard (Clean-to-Random) simulation conditions. The 40.00% real-world figure covers "three real-world tasks." The abstract does not identify which tasks, which baseline model, or who ran the comparison.
Data, training code, and weights are released on GitHub alongside a pretrain dataset on Hugging Face. It lands in a steady run of open-weights robotics work; our robotics tracker has logged 198 stories in the last 90 days.
Originally reported by arxiv.org
Read the original article →Original headline: SJTU and Noematrix's SWAM Scales World-Action Model on 47K Hours of Video, 40% vs 23% Real-World Baseline