ETH Zürich's VPM Jointly Forecasts Video and 3D Point Trajectories

A world model trained to forecast multi-view video and persistent 3D point trajectories jointly beats a video-only counterpart on both pixel fidelity and 3D motion accuracy, researchers from ETH Zürich, Seoul National University, KAIST and NYU report in a new preprint on Hugging Face introducing Video-Point Model (VPM). The system extends the Cosmos 3 backbone with camera-conditioned multi-view processing and a trajectory head that predicts how physical points move through time.

The design turns 3D geometry into a first-class training signal rather than something the model has to infer from pixels alone. In the authors' own phrasing: "supervising these trajectories alongside video generation therefore anchors appearance changes in every camera to a shared description of 3D motion." Each trajectory tracks a single point across all views.

To train and evaluate it, the group built VPM-Data, pairing synchronized multi-view videos with persistent point tracks (8,000 tabletop-manipulation clips with three cameras and 2,000 urban-dynamics clips with four cameras), plus VP-WorldBench, a forecasting benchmark spanning simulated and real environments including the DROID real-robot platform and a real indoor scene captured with 48 cameras. Robot actions can be bolted on as a token modality denoised alongside future videos, "thereby grounding the policy in the same 3D-aware representation" as the generator.

The paper reports that on VP-WorldBench and existing 4D generation benchmarks, point supervision improves both video fidelity (FVD) and 3D motion accuracy (δ³D_avg) over a video-only variant, and in simulated and real-robot experiments it also raises the manipulation success of the action-enabled model. The abstract itself publishes no per-task deltas, so the size of those gains sits in the full paper.

VPM lands in a dense stretch of robotics coverage we have been tracking (167 prior stories in the last 90 days), part of the broader cluster of recent work arguing that policies and world models should share one 3D-aware representation rather than stack vision and action separately.