KeyFrame-Compass benchmark exposes video model trade-offs

Text-to-video demos are impressive right up until a director tries to actually direct one. The workflow professionals want is keyframe-conditioned: hand the model a sequence of reference images and get a video that hits those beats in order. A new benchmark posted to arXiv, called KeyFrame-Compass, is the first attempt to systematically test whether current video models can actually do that, and the headline result is that none of the nine systems evaluated do it cleanly.

The setup is worth understanding because it is what makes the finding hold up. The authors curated 386 samples spanning three application domains, two video structures, two prompt granularities, two conditioning formats, and four keyframe densities. Rather than a single quality score, they decompose keyframe execution into six metrics covering presence, fidelity, temporal ordering, localization, persistence, and uniqueness, then layer MLLM-based judgments on top for overall video quality. That structure lets them isolate where each model breaks.

Two findings do most of the work. Current models exhibit what the paper calls a clear trade-off between faithful keyframe execution and natural video synthesis, and that trade-off gets worse as the keyframes get denser. On top of that, most open-source models fail to interpret storyboard-grid inputs as temporally ordered keyframe sequences at all, which is the specific input format a lot of animation and ad workflows want to use. In practice that means the grid you hand the model gets treated as a bag of pictures rather than a shot list.

The honest caveat is that a benchmark of 386 samples across many axes is a first pass, not a settled ranking, and the abstract as posted does not break out which of the nine systems performed best on which metric. Take the specific numbers as reported, not as the final scorecard.

The useful signal for anyone building on top of these models is that the next real capability leap in video is not another quality jump. It is closing the gap between what a keyframe input promises and what actually comes out the other side, and the lab or open-source project that solves storyboard-grid ordering first has a genuine product wedge for professional creative tools.