arXiv cs.CL· Man-Ling Sung, Hemanth Kandula, Jeff Ma, William Hartmann, Matthew Snover·· 12 小时前AI 评分25
SAUCE：基于对话蕴含分析的跨语言语音摘要评估流水线
SAUCE: Summary Analysis Using Conversation Entailment
AI 导读
SAUCE 是一个无参考、基于事实的跨语言对话语音摘要评估流水线，通过对话与摘要文本之间的蕴含关系衡量摘要的准确性和事实覆盖度。与黑盒 LLM 评估器和稠密嵌入指标不同，SAUCE 将摘要分数映射到离散、可验证的事实，让用户能定位具体的幻觉或遗漏。初步研究表明 SAUCE 与人类判断高度一致，该工作已被 ACL GEM workshop 接收。
正文 · 原文
Abstract:With the growing need for evaluating Large Language Models (LLMs) and their applications to speech, challenges persist in summarizing and evaluating conversations that lack a clear end goal. We introduce SAUCE -- a reference-free, fact-based evaluation pipeline for cross-lingual conversational speech summarization. It measures the accuracy and the fact coverage of a summary through the entailment between conversation and text. We compare SAUCE against several popular summarization metrics and demonstrate the effectiveness of capturing information loss due to transcription and translation error and identifying broken summaries. Crucially, unlike black-box LLM evaluators or dense embedding metrics, SAUCE is inherently explainable: it maps summary scores to discrete, verifiable facts, allowing users to pinpoint exact hallucinations or omissions. We illustrate how this interpretability helps developers systematically profile LLM behaviors and gives end-users an actionable tool to verify summary accuracy in noisy, real-world conditions. Preliminary investigations show SAUCE strongly align with human judgment.
| Comments: | Accepted and published to the ACL GEM workshop. Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM), pages 364-377, San Diego, California, USA. Association for Computational Linguistics |
| Subjects: | Computation and Language (cs.CL); Audio and Speech Processing (eess.AS) |
| Cite as: | arXiv:2611.06812 [cs.CL] |
| (or arXiv:2611.06812v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06812 arXiv-issued DOI via DataCite (pending registration) |
|
| Related DOI: | https://doi.org/10.18653/v1/2026.gem-main.34
DOI(s) linking to related resources |
Submission history
From: Man-Ling Sung [view email]
[v1]
Wed, 4 Nov 2026 17:25:40 UTC (738 KB)
来源：arXiv cs.CL · arxiv.org