arXiv cs.CL· Yilin Jiang, Fei Tan, Xuanyu Yin, Guanchu Wang, Xia Hu, Yutao Yue·· 3 天前AI 评分36
如何测量并减少 LLM 智能体的不可用回复
Measuring and Reducing Unusable Replies of LLM Agents
AI 导读
研究提出"回复可用性"指标，用读者模型从请求者视角判断智能体回复能否被理解和使用，其与五名标注者多数意见的一致率处于个体标注者区间内。在七个家族八个模型的 480 条回复中，54% 失败，最佳模型仍有 30% 失败，失败多因指向请求者未见材料或使用智能体自造标签。明确回复要求可将失败率从 0.52 降至 0.43，再用次级模型改写回复可降至 0.36，并经人工盲评确认。
正文 · 原文
Abstract:People delegate multi-step work to LLM agents and read only the reply that the agent hands back. Current evaluation scores task completion and the faithfulness of reports, and it leaves open whether the requester can understand and use the reply. We call this property reply usability and measure it with a reader model that takes the requester's position and tests whether the reply can be interpreted from what the requester has seen, is in the requester's language and answers the request. Its agreement with the majority of five annotators lies within the range of the individual annotators. Across eight models from seven families, 54% of 480 replies fail, and the best model still fails on 30%. Failed replies mostly point to material the requester has not seen or use labels that the agent coined during its work. Stating explicit requirements for the reply reduces the failure rate from 0.52 to 0.43 together with errors and omissions, and a secondary model that rewrites the reply reduces it to 0.36, which human annotators confirm in a blind comparison. Reply usability can thus be scored alongside task success and improved with one appended message or one call to a secondary model.
| Comments: | 17 pages, 4 figures, 12 tables |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC) |
| Cite as: | arXiv:2610.19659 [cs.CL] |
| (or arXiv:2610.19659v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19659 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yilin Jiang [view email]
[v1]
Wed, 14 Oct 2026 06:35:49 UTC (168 KB)
来源：arXiv cs.CL · arxiv.org