huggingface.co web signal

Meta's memory agent lifts Claude Sonnet 4.5 by 8.3 points

meta agents ai-business

TL;DR

  • On Terminal-Bench 2.0, a Claude Opus 4.6 memory agent lifts Claude Sonnet 4.5 pass@1 from 37.6% to 45.9%, +8.3 pp.
  • Ablations show selective intervention outperforms exposing the full memory bank, always-on injection, advisor-style guidance without a bank, and Mem0-style retrieval.
  • A Qwen3.5-27B memory agent trained with SFT and GRPO transferred to held-out Terminal-Bench, raising a frozen action agent from 37.6% to 41.1%.

Meta AI's new arXiv paper 'Remember When It Matters' reframes agent memory as an intervention policy rather than passive storage and retrieval. The failure mode the authors name is 'behavioral state decay': the trajectory still contains the requirement or the failed command, but by the time the action agent has to decide, that information no longer shapes the next step.

The setup keeps the action agent untouched. A separate memory agent, Claude Opus 4.6 in the reported runs, sits alongside, maintains a structured bank of status, knowledge, and procedural entries, and at each step chooses between emitting a targeted reminder or staying silent. Across 85 tasks of Terminal-Bench 2.0, Claude Sonnet 4.5 goes from 37.6% to 45.9% pass@1, a +8.3 pp gain. On the 278-task τ²-Bench weighted average, Sonnet 4.5 goes from 55.0% to 61.8%, +6.8 pp. The stronger Opus 4.6 action agent still gains, though less, +2.4 pp on Terminal-Bench and +2.5 pp on τ²-Bench.

The ablations are the useful part. Exposing the full memory bank at every step, a passive context augmentation, helps but trails the selective policy on the domain-balanced macro-average. Always-on injection is competitive on micro but worse on macro. Removing the bank entirely and running an advisor-style second model is unstable, actually hurting the airline domain relative to the baseline. And a Mem0 retrieval layer improves the average but does not match the two-phase policy. Maintaining execution state matters, and so does the decision to stay quiet.

The open-weight sketch is more preliminary. The authors train Qwen3.5-27B as the memory agent with SFT and GRPO while keeping a Qwen3.5-122B-A10B action agent frozen. An untrained 27B memory agent actually hurts SETA reward (0.709 to 0.693); SFT recovers to 0.720 and GRPO reaches 0.734. Transfer to held-out Terminal-Bench moves the frozen action agent from 37.6% to 41.1%. The honest caveat is that this is one paper on two benchmarks, run with a frontier model as the memory brain, and the deployment picture where you pay for a second model call at every memory step is not one the paper resolves. There are no latency figures, no token-overhead accounting, and no reported ratio of silence versus reminder.

What is worth watching is whether 'when to intervene' shows up as a primitive in the agent scaffolds people are shipping. If a small trained memory model can consistently pull weaker action agents toward frontier performance, the cheap-action-model-plus-trainable-memory recipe starts to matter more than raw parameter count.