NPCI-hosted benchmark: 6.5-12% of bank agent 'wins' skipped steps

Between 6.5% and 12.0% of sessions on a new Indian retail-banking agent benchmark are judged successful under outcome-only grading even though the assistant skipped a required step or made an unrequested change. The benchmark, posted to arXiv on September 30 by Krishanu Adhikary and Prashant Devadiga, scores 1,000 Indian retail-banking tasks on procedure in addition to the final records.

The authors' objection to outcome-only testing is practical: "Many required steps, however, leave the records unchanged, so an assistant that skips them can still be judged successful." Their worked example is "obtaining a closure quote before closing a deposit", a step whose omission does not show up in the account record afterward but which the bank's rules require.

The 6.5-12.0% range holds "for every model" tested. The abstract does not name the models. Code is released under the npci GitHub namespace and the dataset is hosted on Hugging Face as NPCI/tau-indian-banking.