PersonaChatBench：用多样化模拟用户评测聊天助手
PersonaChatBench: Evaluating Chat Assistants with Diverse Simulated Users
研究者提出 PersonaChatBench 框架，将同一任务配给多个由私有 persona card 定义的模拟用户，以考察聊天助手在不同用户条件下的表现差异。该框架在六个领域评测了 20 个语言模型，分别评估原生任务结果、用户表达需求的满足程度与模拟器保真度。结果显示任务表现与需求满足度随 persona 和领域变化，强任务表现不一定带来个体需求的满足。
Authors:Zheyuan Deng, Taoran Li, Jingwen Zhang, Yuante Li, Dachuan Zhao, Fulei Zhang, Yunkai Zhang, Zhikun Xu, Yajie Liao, Haichao Zhang, Haodi Hu, Cong Wang, Chen Yang, Siyang Wu, Defu Cao, Xu Hu, Haoming Li, Haixin Wang, Robert George, Weizhi Zhang, Yuchu Wang, Zhejian Peng, Yueying Li, Wanting Jiang, Shushu Liang, Zhiyuan Yu, Yifan Wang, Jintao Huang, Yijun Wang, Yuexing Hao, Xiaomin Li
Abstract:Conversational assistants are expected to help people with diverse backgrounds, preferences, and communication styles. Yet evaluations centered on a single cooperative user provide limited insight into how reliably an assistant serves users with different personas. We introduce PersonaChatBench, a framework for evaluating task competence and responsiveness to diverse user needs. PersonaChatBench pairs the same underlying task with multiple simulated users, each guided by a private persona card that specifies distinct needs, preferences, and interaction behaviors. By holding the task fixed while varying the user, the framework examines how assistant performance changes across user conditions. We evaluate 20 language models across six domains, separately assessing native task outcomes, fulfillment of expressed persona needs, and simulator fidelity. Our results show that task performance and need fulfillment vary across personas and domains. Strong task performance does not necessarily translate into satisfying individual needs, and models' relative strengths in meeting those needs are not consistent across domains. PersonaChatBench makes these differences measurable, revealing gaps obscured by aggregate scores and helping identify which users an assistant serves well and where adaptation is needed. It provides a basis for evaluating consistency across users and guiding the development of assistants that better accommodate diverse needs in everyday applications.
| Comments: | 74 pages, including appendices |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.42060 [cs.CL] |
| (or arXiv:2610.42060v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.42060 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zheyuan Deng [view email]
[v1]
Fri, 30 Oct 2026 17:23:14 UTC (8,543 KB)
来源：arXiv cs.CL · arxiv.org