arXiv cs.AI· Yupeng Cao, Xueqing Peng, Lingfei Qian, Haohang Li, Yangyang Yu, Anke Xu, Wenbo Cao, Yan Wang, Yuechen Jiang, Yuyan Wang, Zhuohan Xie, Yankai Chen, Zehan Li, Jimin Huang, Xue Liu, Jian-Yun Nie, Sophia Ananiadou·· 11 小时前AI 评分45
FINCOSMOS：面向金融智能体的开放世界评测范式
FINCOSMOS: An Open-World Evaluation Paradigm for Agents in Finance
AI 导读
FINCOSMOS 是一个面向金融智能体的开放世界评测范式，模拟用户只按需透露隐藏策略，智能体需自行从 Web 获取数据并在数据截止点下构建回测。在 150 场会话中，四个编程智能体和一个无工具模型均无一场通过验收，模拟用户认可的会话全部未通过程序检查，智能体仅主动问出 54–64% 的未告知细节，人类专家将 88.33% 的失败归因于智能体。
正文 · 原文
Authors:Yupeng Cao, Xueqing Peng, Lingfei Qian, Haohang Li, Yangyang Yu, Anke Xu, Wenbo Cao, Yan Wang, Yuechen Jiang, Yuyan Wang, Zhuohan Xie, Yankai Chen, Zehan Li, Jimin Huang, Xue Liu, Jian-Yun Nie, Sophia Ananiadou
Abstract:Language agents increasingly work for users on the open Web, and finance is where trusting them is hardest. The data must come from the Web, the strategy lives in the user's head, and correctness depends on time, so an agent can fail silently: a strategy the user never asked for, unverified data, or a backtest that uses the future yields a report as convincing as a correct one. Existing benchmarks close the world somewhere, fixing the task, the data, or the timeline, and thereby remove these failures instead of measuring them. We introduce FINCOSMOS, an open-world evaluation paradigm for agents doing financial work. A simulated user holds a hidden strategy sampled from a compositional grammar and reveals it only when asked. The agent receives no data, so it must acquire data from the Web, build its own backtest under a data cutoff, and deliver verifiable results. Grading needs no reference answer. Programs verify point-in-time correctness and recompute every reported metric, the delivered strategy is checked against the hidden specification, and a run is accepted only when the user is satisfied and none of 15 veto checks fails. Across four coding agents and a model without tools in 150 sessions, no session is accepted: every session that the simulated user signed off fails a program check, and agents raise only 54--64\% of the details they were not told. Human experts attribute 88.33\% of failures to the agents. With FINCOSMOS feedback, an agent with persistent memory learned all ten tasks.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computational Finance (q-fin.CP) |
| Cite as: | arXiv:2611.07464 [cs.AI] |
| (or arXiv:2611.07464v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.07464 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yupeng Cao [view email]
[v1]
Thu, 5 Nov 2026 05:19:44 UTC (68 KB)
来源：arXiv cs.AI · arxiv.org