arXiv cs.CL· Bingsen Chen, Boyan Li, Ping Nie, Yuyu Zhang, Xi Ye, Chen Zhao·· 12 小时前AI 评分44
研究揭示 Deep Research Agents 在多轮报告修订中不可靠
Beyond Single-shot Writing: Deep Research Agents are Unreliable at Multi-turn Report Revision
AI 导读
新基准 Mr Dre 将多轮报告修订设为 Deep Research Agents 的新评测维度，包含覆盖全面性、事实性与呈现质量的长文评测协议及人工核验的反馈模拟流程。对五款 DRA 的分析显示，智能体虽能处理大部分用户反馈，却会回退 16-28% 此前已覆盖的内容与引用质量。多轮修订后即使表现最好的智能体仍有明显提升空间，且提示词工程与专用修订子智能体等推理时修复手段难以解决该问题。
正文 · 原文
Abstract:Existing benchmarks for Deep Research Agents (DRAs) treat report generation as a single-shot writing task, which fundamentally diverges from how human researchers iteratively draft and revise reports via self-reflection or peer feedback. Whether DRAs can reliably revise reports with user feedback remains unexplored. We introduce Mr Dre, an evaluation suite that establishes multi-turn report revision as a new evaluation axis for DRAs. Mr Dre consists of (1) a unified long-form report evaluation protocol spanning comprehensiveness, factuality, and presentation, and (2) a human-verified feedback simulation pipeline for multi-turn revision. Our analysis of five diverse DRAs reveals a critical limitation: while agents can address most user feedback, they also regress on 16-28% of previously covered content and citation quality. Over multiple revision turns, even the best-performing agents leave significant headroom, as they continue to disrupt content outside the feedback's scope and fail to preserve earlier edits. We further show that these issues are not easily resolvable through inference-time fixes such as prompt engineering and a dedicated subagent for report revision.
| Comments: | ACL 2026 (SAC Highlight Award). 43 pages. Code and data: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2601.13217 [cs.CL] |
| (or arXiv:2601.13217v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2601.13217 arXiv-issued DOI via DataCite |
|
| Journal reference: | Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13325-13356, San Diego, California, July 2026 |
| Related DOI: | https://doi.org/10.18653/v1/2026.acl-long.609
DOI(s) linking to related resources |
Submission history
From: Bingsen Chen [view email]
[v1]
Mon, 19 Jan 2026 16:48:45 UTC (459 KB)
[v2]
Mon, 2 Nov 2026 08:51:59 UTC (473 KB)
来源：arXiv cs.CL · arxiv.org