arXiv cs.AI· Shengjie Ma, Chenlong Deng, Jiaxin Mao, Jiadeng Huang, Teng Wang, Junjie Wu, Changwang Zhang, Jun wang·· 9 天前AI 评分30
Proof-of-Use：缓解深度研究智能体中的工具调用作弊
Proof-of-Use: Mitigating Tool-Call Hacking in Deep Research Agents
AI 导读
针对深度研究智能体在强化学习中出现的 Tool-Call Hacking 问题，研究者提出证据锚定的 RL 框架 Proof-of-Use（PoU），要求智能体可审计地引用规范化证据标识符。
正文 · 原文
Abstract:While reinforcement learning (RL) enhances their ability to plan and reason across retrieval steps, we identify a critical failure mode in this setting: Tool-Call Hacking. Unlike execution-based tools (e.g., code or math), whose effects are directly observable, the weak observability of causal dependencies between retrieved evidence and reasoning under format- and outcome-level supervision enables agents to maximize surface-level reward signals without genuinely grounding their reasoning in the returned evidence. This leads to distinctive pathologies, including mode collapse via tool overuse and hallucinated tool usage where tool calls are largely decorative.
To address this issue, we propose Proof-of-Use (PoU), an evidence grounded RL framework that explicitly optimizes the causal dependency from retrieval to reasoning and final answers. PoU re-fomulate a fine-grained stepwise interaction protocol in which agents must auditably cite normalized evidence identifiers. We operationalize this via a multi-objective reward design consisting of: (1) two progressive process rewards that constrain citation validity at intermediate steps; (2) a global Answer--Support Alignment reward that enforces consistency between final answers and retrieved evidence; and (3) a curriculum-style adaptive reward mixing mechanism that smoothly transitions agents from dense process supervision to sparse outcome-based objectives. Extensive experiments show the strong performance of PoU and demonstrate the effectiveness in mitigating tool-call hacking. Beyond this, PoU exhibits a notable emergent property: adaptive and robust tool-usage patterns naturally arise under domain and tool shifts, even though PoU does not explicitly optimize for tool adaptation.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2510.10931 [cs.AI] |
| (or arXiv:2510.10931v3 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2510.10931 arXiv-issued DOI via DataCite |
Submission history
From: Shengjie Ma [view email]
[v1]
Mon, 13 Oct 2025 02:45:37 UTC (343 KB)
[v2]
Fri, 23 Jan 2026 05:07:44 UTC (620 KB)
[v3]
Thu, 8 Oct 2026 02:04:53 UTC (620 KB)
来源：arXiv cs.AI · arxiv.org