arXiv cs.CL· Raffi Khatchadourian·· 13 小时前AI 评分38
DFAH-Bench：金融智能体的决策一致率掩盖了什么
DFAH-Bench: What Agreement Leaves Unmeasured in Financial Agents
AI 导读
DFAH-Bench 通过重放实验发现，金融智能体重复决策的一致率达 94-95%，但包含输入与结果的工具路径一致率仅 45-52%。在 48 次执行的诊断中，从重复记录里删除相同调用后一致率反而完美，说明一致率会掩盖执行变化与系统性遗漏。
正文 · 原文
Abstract:When a financial agent repeats its decision, how much does that tell us about the actions behind it? DFAH-Bench examines this question through replay experiments, a controlled test of omitted tool calls, and banking tasks with authorization checks. Across 570 eligible prospective replay episodes, decisions agreed at rates of 94-95%, while tool paths, including inputs and results, agreed at 45-52%; one condition fell one group short of the planned minimum. In a separate 48-execution diagnostic, deleting the same calls from repeated records left agreement perfect. In banking probes, repeated authorization decisions sometimes conflicted with constructed rule labels. The $\tau$-Knowledge banking study scheduled 1,080 episodes, with 1,033 known outcomes. Structural checks alone completed more primary tasks than either model-based gate-and-recovery bundle. Input limits and recovery behavior materially shape these results. The typed-choice bundle had lower mean episode cost than the generative bundle on complete task pairs, but fewer successes across the primary schedule under every assignment of unknown outcomes. Missing outcomes prevented the planned confirmatory tests, so the banking comparisons are descriptive. Together, the findings show that agreement alone cannot establish reliability: it can conceal changes in execution, systematic omissions, and unsupported decisions. Evaluation therefore needs to connect what the agent did, the evidence and rules governing its action, and whether it completed the task.
| Comments: | 28 pages, 6 figures; supersedes v3; incorporates the AI4F and SLM-Agents workshop material |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| ACM classes: | I.2.7; I.2.6; J.1 |
| Cite as: | arXiv:2607.20491 [cs.AI] |
| (or arXiv:2607.20491v4 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.20491 arXiv-issued DOI via DataCite |
Submission history
From: Raffi Khatchadourian [view email]
[v1]
Wed, 10 Jun 2026 03:31:09 UTC (67 KB)
[v2]
Fri, 24 Jul 2026 14:53:06 UTC (68 KB)
[v3]
Tue, 22 Sep 2026 00:49:53 UTC (184 KB)
[v4]
Tue, 20 Oct 2026 17:06:32 UTC (128 KB)
来源：arXiv cs.CL · arxiv.org