Claude Opus 5 Leads BizBench Payments Agent Test at 64%

A new agent benchmark grounded in a payments company's own procedures finds that even frontier models stall below two-thirds success once scoring punishes mid-execution rule breaks, not just final outcomes. In BizBench, Claude Opus 5 leads at 64.0% strict success and GPT-6 Sol trails at 61.3%.

The paper wires up 789 knowledge documents and 601 discoverable business tools behind 50 multi-turn tasks involving resource constraints, temporal dependencies, prior commitments, and "conflicting or outdated guidance." Each task is validated by reference execution, with deterministic checks on both goal attainment and execution validity. The authors argue outcome-only scoring misses what matters in regulated work: "intermediate violations can incur financial losses or regulatory consequences regardless of the final outcome."

The gap between the two metrics is the paper's sharpest finding. For Qwen3.8-27B, goal attainment reaches 42.0% but strict success collapses to 32.7%. The authors frame this as evidence of "how outcome-based evaluation can mask invalid execution."

Tool-call overhead is the other number that stands out. DeepSeek-V4.1-Flash averages 191.3 tool calls per run, "approximately 70% more than reference executions."

The abstract names no per-task breakdown, no violation categories, and no identifier for the payments company whose procedures it drew on.