arXiv cs.CL· Tiasa Singha Roy, Manya Wadhwa, Greg Durrett·· 2 天前AI 评分42
LLM 研究智能体如何“平滑”论断：SmoothScore 量化报告过度自信
Quantifying Claim Smoothing by LLM Research Agents
AI 导读
LLM 研究智能体生成的长篇研究报告常把多样甚至冲突的来源压缩成单一叙事，得出比引用证据更果断的结论，研究者将这一失败模式称为 smoothing。为此提出论断级评分流程 SmoothScore，先抽取报告中的关键结论性论断，再检索相关证据并按高层方面聚类，最后逐条打分量化平滑程度。在 7 个系统和 4 个数据集上测试，智能体一致地给出强于证据支撑的论断。
正文 · 原文
Abstract:Deep research, or any use of large language models (LLMs) to read a large document collection and produce insights, requires synthesizing information from different sources. In this work, we find that LLMs produce long-form research reports that read as confident syntheses but which fail to accurately reflect the underlying documents. We call this failure smoothing. In collapsing diverse and potentially conflicting sources into a single narrative, the model discards the evidential "jaggedness" of its corpus and asserts conclusions more decisive than the citations can support. We introduce a claim level scoring pipeline, SmoothScore, to quantify this effect. Our pipeline extracts key conclusive claims from model generated reports, then finds evidence relevant to each claim and clusters that evidence along high level aspects. Finally, each claim is scored with respect to the evidence to quantify the degree of smoothing done by the agent. Across 7 systems and 4 datasets, we find that agents consistently produce stronger claims than warranted.
| Comments: | Code and data present at: this https URL |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.35793 [cs.CL] |
| (or arXiv:2610.35793v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.35793 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tiasa Singha Roy [view email]
[v1]
Mon, 26 Oct 2026 17:29:48 UTC (3,217 KB)
来源：arXiv cs.CL · arxiv.org