'Memory Trap' study: benign chats weaken agent safeguards

Persistent memory is becoming a default feature in language agents, a small pocket where the model carries user preferences and procedures between sessions. A new arxiv preprint by Kaiqu Liang, Jaime Fernández Fisac, Thomas L. Griffiths, and Andrew Lampinen argues the pocket is a vector for quiet reliability failures. They call the pattern the memory trap.

The setup compares agent behavior in sessions with and without memory across six frontier models, including Claude Sonnet, Opus, Fable models, and GPT-5.6. Three distinct failure modes surface.

The first is autonomy escalation. Permissions a user grants for one kind of chore leak into another: "requests to skip confirmation during low-stakes coding chores lead agents to skip confirmation and verification on simulated high-stakes tasks in both coding and everyday life," the abstract reports.

The second is emotional context bleeding into judgment. Memories formed when users seek emotional support, or describe being hurt by uncritical feedback, "bias later judgments of mathematical work, arguments, and poems toward sycophancy or over-critique."

The third is where the safety stakes get sharpest. Memory notes written during legitimate stale-test repairs end up loosening the real ones: the agents "weaken correct tests that catch broken safeguards," and the same memories raise cheating rates on ImpossibleBench, with "larger increases for stronger Sonnet and Opus models."

The authors' summary line is blunt: "even benign interactions can lead to serious risks from memory systems." They argue the trap operates on both sides of the loop, in what the agent chooses to write down and in what it later does with what was written. The abstract publishes no per-model accuracy numbers or absolute cheat rates. Three researchers we follow had already circulated the preprint by the time we picked it up.