HABIT-Bench: best memory method hits 33.61% on user habits
TL;DR
- The best of eight memory methods achieved 33.61% macro accuracy across food, finance, software, and travel domains on four-choice habit tasks.
- Performance stayed limited even when the benchmark supplied agents with annotated evidence of the user's habits directly.
- The authors tested Qwen3-235B-A22B and GLM-5.3-Flash on synthetic histories spanning hundreds of sessions.
The best memory method the authors tested got 33.61% macro accuracy on four-choice tasks that ask whether an AI agent can infer a user's habits from past sessions and apply them to a new request. The benchmark, HABIT-Bench, works across four domains: food, finance, software, and travel.
Eight memory methods were evaluated with two underlying models, Qwen3-235B-A22B and GLM-5.3-Flash. The tasks go beyond simple recall; they include "temporary exceptions, changes in habits, and suggestions that users have not accepted."
The authors' framing is blunt. "Retaining past interactions alone does not ensure that agents can infer habits and apply them appropriately," they write, and performance stayed capped "even when annotated evidence is supplied directly." They describe the result as "a persistent gap between access to past interactions and the ability to infer and appropriately use user habits."
Originally reported by paper
Read the original article →Original headline: HABIT-Bench: top agents cap at 33.6% inferring user habits from past sessions