IndicBankBench: banking LLMs reliable 43.7–58.2% under pass^3

Banking assistants deployed for Indian retail customers pass an all-trials reliability bar on only 43.7% to 58.2% of test cases, according to a new benchmark paper from Suvradip Paul, Chandra Bhushan, Harsh Sharma, Nitin Kukreja, Yatharth Dedhia, Keyur Doshi, and Prashant Devadiga. On the more forgiving at-least-once metric, the same eleven models score 60% to 74%.

The benchmark, IndicBankBench, runs 799 cases across five operational domains, a capability/refusal domain, and twenty primary axes. Each case is scored at four stages: safety, action and tool use, response adequacy, and advisory quality. Tool use and most safety checks are deterministic; a narrow resolver handles only ambiguous confirmation-before-write cases, and a separate LLM judge evaluates semantic response adequacy.

The authors' core argument is that final-response scoring misses the errors that matter in banking. "An assistant may ask for information it already has, rely on stale context, select the wrong account, or write an invalid value after stating the correct one," they write. They run every case three times and report strict pass^3, which requires success on all trials.

"This gap shows that at-least-once success can overstate dependable banking behavior," the paper says. The authors add that case-level diagnostics also distinguish systems that ask unnecessary questions from those that act but fail to reconcile customer context or fully resolve the request.

The abstract publishes no per-model numbers. The authors release the cases, mock environment, and evaluation harness.