arXiv cs.CL· Shengyue Guan, Jindong Wang, Jiang Bian, Bin Zhu, Jian-guang Lou, Haoyi Xiong·· 14 小时前AI 评分34
综述：多轮对话中 LLM 智能体的评估方法
Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey
AI 导读
一篇综述用类 PRISMA 框架系统梳理近 250 篇文献，提出两套互相关联的分类体系：一套界定多轮对话 LLM 智能体"评估什么"，涵盖任务完成、回复质量、用户体验、记忆与上下文保持、规划与工具集成；另一套说明"如何评估"，将方法分为标注评估、自动指标、人机混合策略与 LLM 自评判，并覆盖 BLEU、ROUGE 等传统指标。
正文 · 原文
Abstract:This survey examines evaluation methods for large language model (LLM)-based agents in multi-turn conversational settings. Using a PRISMA-inspired framework, we systematically reviewed nearly 250 scholarly sources, capturing the state of the art from various venues of publication, and establishing a solid foundation for our analysis. Our study offers a structured approach by developing two interrelated taxonomy systems: one that defines \emph{what to evaluate} and another that explains \emph{how to evaluate}. The first taxonomy identifies key components of LLM-based agents for multi-turn conversations and their evaluation dimensions, including task completion, response quality, user experience, memory and context retention, as well as planning and tool integration. These components ensure that the performance of conversational agents is assessed in a holistic and meaningful manner. The second taxonomy system focuses on the evaluation methodologies. It categorizes approaches into annotation-based evaluations, automated metrics, hybrid strategies that combine human assessments with quantitative measures, and self-judging methods utilizing LLMs. This framework not only captures traditional metrics derived from language understanding, such as BLEU and ROUGE scores, but also incorporates advanced techniques that reflect the dynamic, interactive nature of multi-turn dialogues.
| Comments: | Companion repository: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2503.22458 [cs.CL] |
| (or arXiv:2503.22458v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2503.22458 arXiv-issued DOI via DataCite |
|
| Journal reference: | ACM Transactions on Intelligent Systems and Technology 17(4), 1-40 (2026) |
| Related DOI: | https://doi.org/10.1145/3793671
DOI(s) linking to related resources |
Submission history
From: Shengyue Guan [view email]
[v1]
Fri, 28 Mar 2025 14:08:40 UTC (1,847 KB)
[v2]
Mon, 5 Jan 2026 15:14:04 UTC (1,107 KB)
[v3]
Sun, 25 Oct 2026 13:10:49 UTC (1,107 KB)
来源：arXiv cs.CL · arxiv.org