NUS HINT-WAM Lifts LIBERO-90 Success From 78.88% to 85.18%
TL;DR
- HINT-WAM post-trains a world action model using the demonstration's future video as a 'hindsight' signal for a self-teacher, with no rewards or simulator.
- On 90 unseen LIBERO tasks, success rises from 78.88% (SFT) to 85.18%, a 6.30% average gain across 90 tasks at paired t=3.58.
- The method distills with an energy score on the student's own denoising trajectory and beats velocity-matching distillation (79.02%) by 6.16%.
A robot policy can be trained to copy a smarter version of itself, where the smarter version is allowed to peek at the future frames of the demonstration video. That is the single idea behind HINT-WAM, posted on arXiv on Nov. 19 by Phuong Tuan Dat, Qi Li and corresponding author Xinchao Wang of the National University of Singapore.
The headline numbers are on LIBERO-90, 90 manipulation tasks the released Fast-WAM checkpoint has never seen. Supervised fine-tuning gets to 78.88%. HINT-WAM gets to 85.18%, "a gain of 6.30% over SFT on average across the 90 tasks (paired t=3.58)." It also beats full SFT — which fine-tunes the 6.02B-parameter video and action experts together — by 3.94% while training only the action expert.
The mechanism is specific. The same network plays teacher and student. The teacher sees the demonstration's future video through a small hindsight encoder and gated cross-attention adapters that add about 13M parameters, 0.2% of the model. The student only sees deployment-time inputs. Crucially, the student is supervised on the states produced by its own denoising sampler, not on noised ground-truth actions. The authors write that, in their framing, "the future frames of a demonstration, which we call _hindsight_, carry privileged information about the intended task progression."
The distillation objective matters too. Regressing the teacher's velocity field lands at 79.02%, essentially SFT. Swapping it for the energy score, a strictly proper scoring rule that matches distributions from one teacher sample, lands at 85.18% — 6.16% higher, with 48 task wins to 17 losses.
On RoboTwin 2.0, where the released checkpoint had already seen the training data, headroom is thin: HINT-WAM beats the BC control by 1.68% in the clean setting and 1.60% in the randomized setting. Catastrophic forgetting also persists without replay — the four original LIBERO suites drop from 96.55% to 44.05% average success after post-training on LIBERO-90, though that is better than the 37.45% SFT retains and the 32.85% for full SFT.
The paper lands into a dense week of robotics-adjacent research on our wire; it is now the 216th robotics story we have tracked in the last 90 days. The authors note the obvious next step themselves: "closed-loop data collection is a natural next step," since the on-policy states today come from the sampler rather than a running robot.
Originally reported by huggingface.co
Read the original article →Original headline: NUS HINT-WAM Uses Future-Video Hindsight to Self-Distill World Action Models Without Reward Shaping