arXiv cs.CL· Md Tahmid Rahman Laskar, Gundeep Singh, Xue-Yong Fu, Karol Chang, Kevin Sanders, Shi Zong, Tania Habib, Julien Bouvier Tremblay, Shayna Gardiner, Harsh Saini, Matthias Lee, Elena Khasanova, Quinten McNamara, Shashi Bhushan TN·· 3 天前AI 评分28
评估 LLM 作为真实客服技能中的智能体工作流执行器
Evaluating LLMs as Agentic Workflow Executors in Real-World Customer Support Skills
AI 导读
研究构建了一套基于真实客服对话记录的 Skill LLM 基准，通过 MCP 兼容 mock 模拟环境状态并生成多轮工具调用对话，对 17 个模型进行零样本评测。GPT-5.4 大幅领先，QWEN3.5-9B 是 20B 以下最强开源权重模型，在每位评审下均超过多个闭源基线。10 组评审两两一致率平均 Kappa=0.524，表明评审可靠性需与模型能力同等重视。
正文 · 原文
Authors:Md Tahmid Rahman Laskar, Gundeep Singh, Xue-Yong Fu, Karol Chang, Kevin Sanders, Shi Zong, Tania Habib, Julien Bouvier Tremblay, Shayna Gardiner, Harsh Saini, Matthias Lee, Elena Khasanova, Quinten McNamara, Shashi Bhushan TN
Abstract:Enterprise customer-support agents increasingly rely on Large Language Models (LLMs) to execute bounded, tool-using workflows. We study this problem through the lens of Skill LLMs: specialized executors that operate within a selected skill context and must complete a narrow, stateful workflow under policy and tool constraints. We present a benchmark construction pipeline that mines recurrent workflows from real-world transcripts, converts them into structured skill descriptions, maps them to tool requirements, simulates environment state with Model Context Protocol (MCP)-compatible mocks, and generates multi-turn tool-using dialogues. Using this benchmark, we conduct a large-scale zero-shot evaluation of 17 language models spanning closed model families and deployment-oriented open-weight models from 1.7B to 14B parameters, each judged independently by five strong LLM judges, plus a deterministic rule-based evaluator that checks tool coverage, hallucinated tool use, and unsupported claims. We find a consistent ranking in which GPT-5.4 leads by a wide margin and QWEN3.5-9B is the strongest open-weight model (below 20B), outperforming several closed baselines under every judge. We also find that judge choice is not a negligible methodological detail: pairwise judge agreement on the pass/fail verdict averages Kappa=0.524 across 10 judge pairs. These results suggest that comprehensive zero-shot evaluation of skill execution requires judge reliability as carefully as model capability.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.14346 [cs.CL] |
| (or arXiv:2610.14346v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14346 arXiv-issued DOI via DataCite |
Submission history
From: Md Tahmid Rahman Laskar [view email]
[v1]
Fri, 9 Oct 2026 21:45:51 UTC (202 KB)
[v2]
Wed, 14 Oct 2026 16:00:21 UTC (203 KB)
来源：arXiv cs.CL · arxiv.org