Slot-JEPA Lifts Robot Planning to 86.1% From V-JEPA 2.1's 35.2%

A team from the Chinese Academy of Sciences has published Slot-JEPA, which takes a frozen V-JEPA encoder, factorizes its dense patch features into sixteen stable semantic slots plus time-varying spatial masks, and uses that interface to plan robotic-arm trajectories. On the authors' REALM-JEPA benchmark the full stack reaches 86.1% overall success (89 of 108 episodes) against 35.2% for V-JEPA 2.1 and 58.3% for V-JEPA 2.1 paired with the paper's own Transition Memory Predictor.

The compression is aggressive. The factorized representation uses roughly 2.5% the scalars of the dense patch field it replaces, and the paper reports that slot-reconstructed features still hit 56.3% mIoU on COCOStuff-27 against 56.0% for V-JEPA 2.1, and 82.4 J-Mean on DAVIS-2017 against 81.9. The planning gains come mostly from the harder sub-tasks: Pick jumps from 23.3% to 76.7%, Rotate from 20.0% to 83.3%, Place from 0.0% to 76.7%.

A second ingredient is the loss function. The authors replace pointwise ℓ₁/ℓ₂ distances with an Unbalanced Optimal Transport cost between aligned masks, solved via Sinkhorn iterations. They report Spearman correlations between predictor loss and real end-effector-to-goal distance of 0.832 for the observed loss and 0.705 for the predicted loss, versus 0.547 and 0.489 for V-JEPA 2.1.

The paper is candid about what the slot interface does not guarantee. The authors write that "slots may merge multiple objects or split one object" and that there is "no guaranteed instance- or part-level decomposition." The memory module, which samples past transitions by relevance, is similarly flagged: fixed state and mask cues "may miss contact/appearance changes."

The arrival lands in a dense week for robotics world-model research on our tracker, including ETH Zürich's VPM and Meta and Oxford's OverSeer, both posted the same day. Code and weights are at github.com/KID-7391/Slot-JEPA.