RCORE curbs object-label shortcuts in video action models

A quiet arxiv preprint this month poked at something worth paying attention to in video AI: what we call 'action recognition' is often not really watching the action. In Why Can't I Open My Drawer?, the authors argue that zero-shot compositional models, the ones expected to recognise novel verb-object pairs they never saw in training, are quietly cheating. They predict the verb by looking at the labeled object, not by attending to motion in the frames.

To pin the failure down the paper proposes diagnostic metrics, including False Seen Prediction, and shows that existing methods overfit to the object-verb pairs they saw during training. Evaluation runs on two compositional benchmarks. One is Sth-com, containing 79K videos with 161 verbs and 248 objects. The other is a new dataset the authors introduce called EK100-com, with 71K egocentric videos, 81 verbs and 216 objects, where the label coverage ratio is only 7.5 percent, deliberately harder than the 12.8 percent in Sth-com.

Their proposed fix, RCORE, has two pieces. Co-occurrence Prior Regularization penalises the model for leaning on frequent object-verb pairings by treating them as hard negatives. Temporal Order Regularization for Composition pushes the model to be sensitive to the order of frames, so a verb representation actually depends on watching motion unfold. On Sth-com with a CLIP backbone the authors report RCORE improves the verb@unseen-comp by 4.6 points and the unseen composition accuracy by 3.8 points. On EK100-com the reported unseen composition accuracy gain is 6.9 points with CLIP and 7.0 points with an InternVideo2 backbone.

Why this matters if you are not writing computer vision papers: the assumption that a model 'understands' actions from video underpins a growing pile of deployed systems, from content moderation to robotics policy learning to safety monitoring. If your model in production has been graded on splits where the same objects and actions co-occur as in training, its accuracy number may partly be a co-occurrence hit rate wearing a costume.

The honest caveats are that this is an arxiv preprint with no confirmed peer-reviewed venue, the reported gains are on two academic benchmarks, and the paper does not spell out inference cost or robustness on noisy out-of-domain video. What it does hand you cheaply is a diagnostic, the false-seen and false-co-occurrence metrics, that any team shipping a video action model could run against their own eval before treating the headline score as understanding.