Study: knowledge graphs beat notes when agents change models

Compressed agent memory does not survive a model upgrade cleanly, according to a controlled study by Ankit Goyal and Jaideep Ray posted to arxiv.org. Their paper tests four memory storage approaches across model transitions and finds that fixed-schema knowledge graphs hold accuracy nearly steady, with changes of ±0.002, while compressed natural-language notes swing by nine to thirteen percentage points in either direction.

The setup was deliberately small: 48 synthetic histories, two open-weight models under 10 billion parameters, and exact-match scoring. Store-only repairs failed in all 48 test cases, the paper reports, while retaining raw source histories enabled successful recovery in 34 of them.

Retrieval told a similar story. The authors write that "partial embedding migrations using a 50/50 mixed index capture only a 4.96-point accuracy improvement," versus 11.90-point gains for full re-embedding. And 81% of retrieval-augmented generation failures were traced to retrieval problems rather than generation, while 80% of note-based accuracy loss happened during the initial compression step, before any model swap.

Two researchers on our Who's Who list posted the paper soon after it appeared. The study stops short of prescribing a memory format for production agents, but the gap between structured schemas and compressed prose is the number it keeps coming back to.