arXiv cs.CL· Ravil Mussabayev, Saveliy Zavertyaev, Dmitriy Parpulov, Andrey Savchenko·· 13 小时前AI 评分38
SIT-Eval：基于情境的对话智能体逐轮交互评估框架
SIT-Eval: Situation-Based Interactive Turn-by-Turn Evaluation of Conversational Agents
AI 导读
SIT-Eval 是一个可调节的对话智能体评估框架，将期望行为形式化为“若情境为 P，则遵循规则 Q”的情境规则，由 LLM 对话方在每轮对话中执行均匀采样的情境，再由上下文感知的 LLM 裁判依据所有条件适用的规则评分。在 1292 个对话轮次上评测 8 个 LLM 发现，总体得分相近的模型在单条规则遵循度上差异很大，所有模型在歧义澄清与苏格拉底式提问上表现最差。
正文 · 原文
Abstract:The rapid proliferation of conversational AI agents necessitates evaluation methods that assess performance across diverse dialogue scenarios, utilizing multiple situational criteria, as appropriate behavior depends on context and encompasses various aspects. Existing dialogue benchmarks typically use static natural conversations, which underrepresent rare situations---a critical safety gap, as low-probability events become frequent failures at production scale. However, limited sample sizes force a choice: either maintain natural dialogues for realism or turn to adaptive situation sampling for comprehensive yet statistically reliable coverage. We introduce SIT-Eval, an adjustable framework for evaluating situation-based conversational agents. We formalize an agent's desired behavior as situational rules: if situation is P, follow rules Q. At each conversation turn, an LLM-interlocutor enacts a uniformly sampled situation. A context-aware LLM-judge leverages all conditionally applicable rules as criteria to evaluate the agent's reply for that situation. The judge is validated using human-verified synthetic golden dialogues. Benchmarking 8 LLMs on 1292 conversation turns reveals failure modes hidden by aggregate metrics: models with similar overall scores vary widely in their adherence to individual rules. All models struggle most with ambiguity clarification and Socratic questioning.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.23612 [cs.CL] |
| (or arXiv:2610.23612v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.23612 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ravil Mussabayev [view email]
[v1]
Fri, 16 Oct 2026 15:24:02 UTC (759 KB)
来源：arXiv cs.CL · arxiv.org