τ-bench-LILT：评估对话智能体在多语言真实工具使用任务上的表现

{\tau}-bench-LILT: Evaluating Conversational Agents on Realistic Multilingual Tool-Use Tasks

AI 导读

τ-bench-LILT 是 τ³-bench 的多语言扩展版本，覆盖德语、韩语、土耳其语和波斯语，涉及航空、零售、电信和银行四个领域。该基准对客户数据库、用户模拟器提示词和任务均做了本地化处理，在八个智能体模型上的评测显示，韩语和波斯语零售领域的 pass^4 相比英语最多下降 61 分。代码与数据已开源。

正文 · 原文

View PDF HTML (experimental)

Abstract:Frontier evaluations of tool-using conversational agents are conducted almost exclusively in English, leaving a critical gap for organizations deploying agents in other languages and regions. We present ${\tau}$-bench-LILT, a multilingual extension of ${\tau^3}$-bench covering German, Korean, Turkish, and Farsi in the airline, retail, telecom, and banking domains. We natively localize every layer of the benchmark: customer databases are populated with culturally appropriate records, user-simulator prompts are refined iteratively through contrastive annotation by native-language experts, and tasks are updated according to the localized database and to reflect realistic tasks in the target region. Evaluation across eight agent models shows the largest drops compared to English in the Korean and Farsi retail domains, by up to 61 points in pass^4. Code and data released at this https URL.

Comments:	17 pages, 14 figures
Subjects:	Computation and Language (cs.CL)
ACM classes:	I.2.7; I.2.11
Cite as:	arXiv:2611.07203 [cs.CL]
	(or arXiv:2611.07203v1 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2611.07203 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Kaden Uhlig [view email]
[v1] Wed, 4 Nov 2026 22:47:38 UTC (9,520 KB)

来源：arXiv cs.CL · arxiv.org