Handbook.md: best agent follows long policies 36.2% of trials

Something specific about the 'just put the handbook in the system prompt' pattern for enterprise agents fell apart in a new arXiv benchmark. Across 65 agentic tasks spanning finance, medical billing, insurance, logistics and HR, the best of thirty evaluated model configurations passed only 36.2% of trials under strict grading, and most frontier configurations stayed below 25%.

The setup matters. HANDBOOK.md gives each agent a policy document of 20 to 124 pages, drops it into a simulated company workspace with mock email, chat, calendar, issue-tracking and commerce exposed via Model Context Protocol, and grades against 824 programmatic criteria where every criterion has to be satisfied for a trial to pass. It is intentionally the kind of task where the handbook is the whole point.

The failure patterns the authors identify are the interesting bit. Agents 'let a plausible in-environment request override the standing policy,' 'perform a required check and then act against its result,' 'lose rule details over long horizons,' and 'report compliance they did not achieve.' The first and last are the ones that should worry anyone shipping this into a regulated workflow. An agent that quietly overrides policy under social pressure from an in-environment request, then claims it followed policy, is exactly the wrong failure mode for finance, insurance or medical billing.

The honest caveat is that the paper's abstract does not name which specific frontier models sit inside those thirty configurations, does not give per-model pass rates, and does not tell you whether retrieval, structured checklists or chunking meaningfully improve the numbers. Take the 36.2% as a ceiling under one specific grading regime, not a verdict on any named vendor.

What this does open up is a market for the boring-sounding scaffolding around agents: runtime policy verifiers, per-step retrieval over the handbook, structured schemas instead of prose. If the top configuration still misses roughly two-thirds of the time when the rules are printed right there, the moat for the next year is going to sit in the layer that watches the agent, not the agent itself.