SJTU RoboFollow Exposes VLA Instruction-Following Mirage

Fine-tune π₀.₅ on Scene 1 of a new manipulation benchmark and it lands a 99.1% Intent Score. Keep the instruction identical but swap the object positions and the same policy drops to 45.5%. That gap is the central claim in RoboFollow, a diagnostic benchmark from the AutoLab at Shanghai Jiao Tong University and Anyverse Dynamics.

The authors trace the illusion to what they call low "scene entropy": in most manipulation benchmarks, the visual scene already implies the task, so a policy can score highly "while barely using it." RoboFollow deliberately builds training scenes where several kinematically valid tasks share the same layout, and reports its own training entropy at 3.782 bits against 0.880 bits for the equally weighted LIBERO Spatial/Object/Goal/Long suites.

Across nine state-of-the-art VLA and world-action policies — π₀, π₀.₅, NVIDIA's GR00T N1.6, openvla-oft, xvla, ACoT-VLA, Lingbot-VLA, Motus and FAST-WAM — the pattern repeats. π₀ falls from 98.2% at L0 to 0.0% on Scene 1's L1 layout perturbation. π₀.₅ decreases from 100.0% at L0 to 34.2% at L3 on Scene 2. "The consistent drop in Intent Score indicates that the degradation cannot be explained solely by low-level execution failures; failures in semantic intent selection are a major contributing factor," the paper reports. The CARE paper published a day earlier attacks the same layer of the stack from the execution side.

The authors then try the obvious fixes and report they don't work. Swapping in Qwen3-VL-4B, which scored 19/20 on their scene QA probe versus 3/20 for the fine-tuned π₀.₅ backbone, and pairing it with QA co-training in a Qwen-GR00T build, still leaves L1–L3 in "catastrophic collapse." LangForce yields "only marginal changes." Classifier-Free Guidance turns "counterproductive," with a guidance scale of 1.2 and 1.5 degrading even L0. Doubling demonstrations from 25 to 50 per task narrows the L0–L2 Intent gap from 54.9 to 55.8 points, essentially not at all.

A real-robot pilot with π₀.₅ mirrors the simulation: success drops from 20/40 (50%) on training instructions to 6/40 (15%) on held-out instructions over the same object set. The paper leaves structural layout diversity as an untested route forward, warning that training on the existing L1 test layouts would "invalidate the split."