LaMem-VLA hits 97.6% on LIBERO with dual latent memory

Robot manipulation policies have spent the last year in a familiar loop: scale the vision-language backbone, add a diffusion action head, and hope the model figures out the temporal structure of a multi-step task on its own. The LaMem-VLA paper on Hugging Face argues the ceiling on that approach is lower than people think, and that a better place to spend effort is memory design inside the model's own latent space.

The headline numbers are notable if you take them as reported. On the LIBERO benchmark, which bundles five manipulation suites including two long-horizon ones, the authors report a 97.6% average success rate, 1.1 points above MemoryVLA and 4.4 above CogACT. On SimplerEnv-Bridge the model reaches 73.9%, which is 16.6 points above the CogACT baseline the scanner draft highlighted, and 2.0 above MemoryVLA. The mechanism is a dual latent memory: a short-term vault holding visual tokens from the current episode, a long-term vault holding action hidden states across the trajectory, and four modules (Curator, Seeker, Condenser, Weaver) that retrieve and inject those memories directly into the VLA's embedding sequence rather than bolting them onto the action head.

The architecture sits on a 7B-parameter Prismatic VLM with DINOv2 vision and a LLaMA-7B language model, plus a roughly 300M diffusion action expert producing 16-step chunks at 7-DoF. Ablations do the useful work of showing where the gains come from. Removing either memory stream drops SimplerEnv performance materially, and swapping the latent-native design for external policy-side conditioning gives back roughly 2 points, which is the paper's real claim: memory has to live in the same latent space as perception and action, not sit alongside as auxiliary context.

The honest caveat is that everything here is simulated. The authors explicitly note there are no real-world robot experiments yet and plan to add them in a future version. What the reporting also does not give you is inference-time cost against a memory-free baseline, or whether the retrieval design holds up on tasks outside the training distribution. Benchmark wins in LIBERO have not, historically, translated cleanly to physical hardware.

Still, the direction is the useful part to watch. If the pattern of latent-native memory beating external conditioning generalises, teams already building on Prismatic-class 7B VLA stacks have a fairly cheap architectural upgrade path, and the long-horizon suites, the ones closest to real warehouse and household work, are where the gains are largest.