arXiv cs.CL· Sergio Burdisso, Esa\'u Villatoro-Tello, Dairazalia S\'anchez-Cort\'es, Yanis Labrak, Jordi Luque, Milos Cernak, Petr Motlicek·· 2 天前AI 评分31
DialFork：评估对话式智能体程序性幻觉的基准
DialFork: Going Beyond Factuality to Evaluate Procedural Hallucination in Conversational Agents
AI 导读
研究团队推出 DialFork，首个基于原始对话文本衡量"程序性幻觉"的基准——智能体话语流畅且事实合理，却偏离既定对话流程。该基准通过在受控轮次分叉合规对话并注入四类幻觉构建，覆盖 12 个领域 24 项任务共 36,278 段对话，由 30 名人工标注者验证。
正文 · 原文
Abstract:Large language models (LLMs) deployed as conversational agents are increasingly expected to follow procedural logic. While factual hallucination has received extensive attention, a different failure mode has been overlooked: procedural hallucination, in which an agent produces individually fluent and factually plausible utterances while deviating from the intended conversational process. We introduce DialFork, the first benchmark to measure this failure mode on raw dialogue text. DialFork is built by forking conformant dialogues at controlled turn positions and injecting one of four hallucination types, yielding 36,278 dialogues across 24 tasks in 12 domains; 30 human annotators validate the construction. We define three evaluation tasks that characterize procedural deviation at three granularities: how far a set of dialogues deviates from conformant ones (monitoring deployed agents), how far a single dialogue does (procedural retrieval), and at which turn a dialogue first goes off-track (real-time guardrails). For the first two, we propose dialog-flow-aware metrics, FlowKL and FlowScore, that compare dialogues by how they unfold rather than their content. At the dialogue level, FlowScore significantly outperforms all baselines across seven encoders, showing that semantic similarity is no substitute for procedural similarity. We release DialFork under the MIT license at this https URL, hoping the community will measure conversational process alongside content.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.21561 [cs.CL] |
| (or arXiv:2610.21561v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.21561 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sergio Burdisso [view email]
[v1]
Thu, 15 Oct 2026 09:18:32 UTC (2,793 KB)
来源：arXiv cs.CL · arxiv.org