Zisserman group disentangles camera vs object motion in video
TL;DR
- Knobel, Zisserman and Asano's Structured Dynamics Model separates camera motion from object motion using frozen features of a pretrained image vision transformer.
- SDM is trained by self-supervised learning on real video plus weak supervision of scene dynamics on synthetic Kubric data, via future-feature prediction.
- On a new benchmark called ProbeMotion, SDM compares favorably to strongly supervised representations such as VGGT despite using substantially weaker supervision.
A quiet paper from Lukas Knobel, Andrew Zisserman and Yuki M. Asano on arXiv is doing something I find more interesting than the usual video-model release. They take a pretrained image vision transformer, freeze it, and get it to disentangle camera motion from object motion in video without labels for either. The frame-to-frame change in a natural video mixes those two sources of dynamics together, and the abstract's own framing is that this decomposition has stayed underexplored precisely because the two are so tightly coupled and so hard to supervise separately.
Their Structured Dynamics Model, SDM, tries to pull them apart by predicting future features and treating the dominant temporal change as one signal and the residual dynamics as another, rather than compressing everything into one entangled latent or a soup of dense transition tokens. Training is self-supervised on real video, with a weak supervisory signal added from synthetic Kubric scene dynamics. The backbone stays frozen throughout, which is the part that matters for anyone thinking about cost.
The reason to care is the comparison. The authors evaluate on ProbeMotion, a new suite they built spanning synthetic and real videos with camera motion, object motion, and combined dynamics, and report that SDM compares favorably to strongly supervised representations such as VGGT on several probes despite using, in their words, substantially weaker supervision. For teams that assumed motion understanding meant paying for a fully supervised pipeline, that is a nudge worth registering.
The honest caveat is that this is one paper, ProbeMotion is the authors' own benchmark, and the abstract's claim is 'compares favorably', not 'beats'. What the reporting does not give you is per-probe numbers, the identity of the frozen backbone, or evidence the approach holds up on long or messy real-world video. So take the specifics as reported, not settled. But the direction, structured video-dynamics representations lifted out of frozen image models with weak supervision, is where the practical progress in this corner of the field seems to be heading, and robotics, AR, and video-understanding teams are the ones best placed to benefit if it holds up.
Originally reported by paper
Read the original article →Original headline: Oxford's Zisserman Separates Camera from Object Motion in Video Without Labels, Beats Supervised VGGT