LatentStream trades retrieval for evolving latent video memory

The paper's move is a small pivot in vocabulary that carries a bigger technical claim: streaming video memory should stop being a bank you query and start being tokens the model carries forward. The authors, Hongyu Qu, Guangming Yao, Ling Xing and colleagues, in a preprint posted to Hugging Face, call the framework LatentStream and describe it as a shift "from store-and-retrieve to retrieve-and-internalize."

Existing streaming MLLMs, the abstract says, "typically compress historical observations into an external memory bank and retrieve query-relevant evidence as additional visual context." That works, but it "prevents it from being internalized into a compact, evolving latent memory that can continuously guide streaming reasoning."

LatentStream stitches three parts together. A "Query-agnostic Hierarchical Streaming Memory" organizes visual history into short-, mid-, and long-term levels under a fixed memory budget through Jenks-guided adaptive consolidation. When a query arrives, a Hierarchical Latent Memory Evolution module equips groups of latent memory tokens with "progressively expanding memory receptive fields," letting them pull historical evidence from their assigned scopes and fold it into a compact, fixed-length representation. A third piece, Progressive Confidence-guided Latent Memory Optimization, builds a reward from group-wise predictive entropy and jointly refines the memory tokens and the retrieved evidence.

The authors report "new state-of-the-art results on existing online and offline video benchmarks." The abstract names none of them, and no per-benchmark numbers, ablations, or compute figures appear. It arrives alongside a run of streaming-video-memory work AI Weekly has tracked in multimodal, where the pattern of the last quarter has been a steady rewrite of how video MLLMs hold context.