arXiv cs.AI· Dongnuan Tian, Yan Yu, Yangfan Liu, Wenjun Tang, Junting Kan, Xiang Wang, Sicong Liu, Yangyang Zhang, Shuanghong Li·· 4 小时前AI 评分35
面向投资研究的 AI 智能体可靠评估框架:区分结果质量与执行质量
Reliable Evaluation of AI Agents for Investment Research: A Framework for Performance and Validity Assessment
AI 导读
研究者提出一套面向投资研究任务的 AI 智能体评估框架,将结果质量与执行质量分开衡量,并通过任务特定标准、确定性验证器和独立 LLM 评审适配不同任务需求。该框架还引入评估有效性机制,检查评估输入、参考信息、支撑证据与评分输出的完整性,并用专家复核处理评分分歧。数值结果显示,该框架能区分智能体在各评估维度上的表现差异,同时识别评估过程中的潜在可靠性问题,相关实现与评估材料将在 GitHub 发布。
正文 · 原文
Abstract:We consider the problem of evaluating AI agents undertaking investment-research tasks, where agents interact with external information sources and computational tools to produce financial analyses and research reports. An important challenge is that the quality of an agent's final output does not necessarily reflect the quality of its execution process, while the reliability of the resulting assessment also depends on the validity of the evaluation procedure. Furthermore, investment-research tasks differ considerably in their requirements for numerical accuracy, information retrieval, financial analysis and supporting evidence, making it difficult to assess agent performance using a uniform set of criteria. We propose a domain-oriented evaluation framework that distinguishes outcome quality from execution quality and accommodates these differences through task-specific criteria, deterministic validators and independent large language model judges. We also introduce an evaluation-validity mechanism that examines the integrity of evaluation inputs, reference information, supporting evidence and grading outputs, with expert review used to resolve identified deficiencies and significant grading disagreements. Numerical results show that the proposed framework can distinguish differences in agent performance across evaluation dimensions, while identifying potential reliability issues in the evaluation process. The associated implementation and evaluation materials will be made available on GitHub.
| Comments: | 21 pages, 2 figures |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.22338 [cs.AI] |
| (or arXiv:2611.22338v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.22338 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Dongnuan Tian [view email]
[v1]
Tue, 17 Nov 2026 07:40:37 UTC (3,390 KB)
来源:arXiv cs.AI · arxiv.org