HKUST, KlingAI VMem-Bench: Video Memory Hurts Precision

Retrieval increases entity recall from 0.559 to 0.628–0.693 but decreases precision from 0.977 to 0.915–0.938 and forbidden-entity avoidance from 0.968 to 0.845–0.898. That is the headline finding of VMem-Bench, a diagnostic study from researchers at the Hong Kong University of Science and Technology, City University of Hong Kong and KlingAI, which asks whether more visual memory actually makes long-generated video better. It lands into a steady run of video-generation research tracked here over the past three months.

Short answer from the abstract: not reliably. "The intuition is compelling: more access to history should make future video more consistent. But an image can be faithful to the past and wrong for the present," the authors write.

The benchmark runs two tracks. Track-A measures which references four retrieval policies select from 92 real films covering 16,548 segments, before any generator is involved. Track-B drives the same policies through a shared video generator on 30 causally authored stories (2,824 five-second segments). Memory helps on short-gap character returns, where recall climbs from 0.553 to 0.698–0.749. On long-gap returns it looks even better. No-memory recall is already 0.984, until the authors strip appearance descriptions out of the prompts. No-memory recall then collapses to 0.565, and retrieval adds only +0.016, a gain whose 95% confidence interval includes zero.

The present-tense failures are concrete. Avoid-OK for destroyed or deceased entities drops from 0.964 to 0.829–0.889. Dead characters wander back on screen. For characters whose appearance has officially changed in the story, retrieval raises similarity to the old look by 0.15–0.20 while presence of the new description falls by 0.03–0.18. The paper lands plainly: "retrieval relevance or final-video consistency alone does not establish memory utility."