arXiv cs.CL· Krishanu Adhikary, Prashant Devadiga·· 13 小时前AI 评分29
TauIndianBankBench：面向工具调用对话智能体的流程感知、动作导向评测基准
TauIndianBankBench: Procedure-Aware, Action-Oriented Evaluation of Tool-Using Conversational Agents
AI 导读
研究者提出 TauIndianBankBench，一个包含 1000 个印度零售银行任务的动作导向基准，用于评测语言模型助手在模拟客户对话中是否完成所有必需步骤且未做未被请求的改动。仅检查最终记录时，各模型有 6.5%-12.0% 的会话被判成功，尽管助手跳过了必需步骤或做了未被请求的改动。同时评估流程与结果能更准确反映模型在印度零售银行业务中的表现。
正文 · 原文
Abstract:Many Indian banks serve customers through FAQ pages and scripted chatbots. Assistants based on language models could respond faster and carry out requests such as blocking a card, but only if they give correct information and follow the bank's rules. These rules prescribe a procedure, a set of required steps, such as obtaining a closure quote before closing a deposit. Adapting models to this domain requires measuring how well they follow the procedure, before and after training. Benchmarks usually judge only the records left at the end of a conversation. Many required steps, however, leave the records unchanged, so an assistant that skips them can still be judged successful. We present TauIndianBankBench, an action-oriented benchmark of 1,000 Indian retail-banking tasks where an assistant helps a simulated customer. Besides the final records, it checks that the assistant took every required step and made no unrequested change. For every model, 6.5-12.0% of sessions are judged successful when only the final records are checked, although the assistant skipped a required step or made an unrequested change. Scoring both procedure and outcome shows more accurately how well language models perform in Indian retail banking.
| Comments: | 15 pages, 6 figures, 14 tables. Code: this https URL data: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.30161 [cs.CL] |
| (or arXiv:2610.30161v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.30161 arXiv-issued DOI via DataCite |
Submission history
From: Prashant Devadiga [view email]
[v1]
Wed, 30 Sep 2026 15:16:21 UTC (272 KB)
来源：arXiv cs.CL · arxiv.org