UW-Madison's MaLoW stores streaming video history in LoRA weights

A new paper from the University of Wisconsin-Madison and Peking University, posted to Hugging Face, attacks the streaming-video memory problem by refusing to put the history into the context window at all. It writes the history into the model's weights.

The system, MaLoW ('Memory as Weights'), feeds each incoming video chunk through a recurrent HyperNetwork that updates a fixed-size LoRA adapter sitting on a base VLM. 'MaLoW uses a recurrent HyperNetwork to incrementally consolidate incoming video observations into fixed-size LoRA weights, allowing historical information to be carried forward without revisiting the full video stream,' the authors write. The design slogan: 'MaLoW represents history as weights, while keeping the current observation as input.'

The numbers the paper publishes include 84.8% on StreamingBench paired with Qwen3-VL-4B and 74.5% on OVO-Bench paired with MOSS-VL-Real-11B. On OVO-Bench's backward split, the abstract says MaLoW 'outperforms prior memory approaches by 3.9%-11.8% across different video backbones while incurring only modest latency overhead.' Dropped onto a WidowX vision-language-action model on Simpler-Bridge, average success rose from 59.4% to 68.8%, which the abstract reports as a 9.3% gain.

Costs stay small by the authors' accounting. The LoRA adapter adds between 1.4% and 4.0% of the base model's trainable parameters depending on backbone, and the memory mechanism contributes roughly 23 ms of additional latency per step.

To train it, the authors built MaLoW-Instruct, 'a large-scale streaming video dataset containing 381K question-answer pairs over 56K videos.' Thirty-six percent of those QA pairs are deliberately queried after the relevant event has finished, which forces the model to lean on its internal state rather than its current frame.

MaLoW lands amid a visible run of recent work on baking context into weights rather than input; our tracker covered ByteDance Seed's Cortex applying a similar idea to text LLMs earlier this week.