PAST-Bench tests whether personal AI agents actually learn from memory

A new benchmark from a group of academic researchers asks a question the 'AI agent memory' pitch usually skips: when a personal agent retains preferences, past workflows, and corrected facts across sessions, does the next session actually get better because of that retained state, or is the improvement coming from somewhere else? PAST-Bench, released as a paper on Hugging Face, tries to isolate the answer with a matched-ablation design where every evaluation episode runs twice under identical conditions except that persistence is toggled on or off.

The suite covers 26 scenarios and 204 episodes across four capabilities — memory, procedural reuse, information gathering, and update — and evaluates seven base models and four agent frameworks. Every base model gains from persistence, with the family-balanced gap Δ ranging from +0.13 to +0.24, but where each model concentrates that gain varies sharply. GPT-5.4 spreads its movement across Memory and Update, GLM-5.1 puts nearly half of its movement on Update, and Kimi K2.6 puts nearly half on Memory. A single Overall Δ, the authors argue, tells the wrong story.

The more uncomfortable finding is a separate mechanism-evidence score, which asks whether the improvement actually used the intended save-retrieve-update pathway. Two frameworks can hit the same headline Δ=+0.13, but nanobot lands at Mech 0.57 against Hermes's 0.64, meaning the improvement did not consistently pass through a write-then-read trace. Same score, different reasons for the score.

The authors then extend Hermes with five targeted runtime interventions — a plan-time consultation gate, typed memory rendering, skill routing, retrieval gating, and a synchronous closeout — packaged as Hermes+. On MiniMax-M2.7 it raises the mean Overall Δ from +0.13 to +0.15 and Mech from 0.64 to 0.73, with its strongest lift on tasks that require replacing outdated state (Update Δ from +0.12 to +0.24).

The honest caveat sits in the paper itself. The +0.02 Overall improvement is smaller than the run-to-run standard deviation (±0.04 for Hermes, ±0.06 for Hermes+), so it should not be read as a stable aggregate gain. The Update variance also widens, and Hermes+ regresses slightly on DeepSeek-V4-Pro and Claude Opus 4.6 when swapped in unchanged. What the reporting does not give you is any test on real, human-authored user traces over long horizons, or on families that interfere with each other, or the token and latency cost of running all five interventions in production.

The forward-looking take is that this reads less as a leaderboard and more as a diagnostic that framework builders and buyers can actually point at. If a vendor sells 'our agent learns from you,' this is now the kind of matched-condition test that would tell you whether the learning is real, or whether the headline number is being carried by the base model doing what it would have done anyway.