arXiv cs.AI· Xue Li, Hongqian Xia, Daniel Gomm, A\'ecio Santos, Cornelius Wolff, Madelon Hulsebos·· 3 天前AI 评分47
智能体数据科学中的沉默失败：现有评测为何掩盖了方法论缺陷
The Silent Failures in Agentic Data Science
AI 导读
研究指出，现有智能体数据科学（agentic DS）评测只看下游预测性能和代码正确性，掩盖了数据泄漏、验证不当等方法论失败。作者在 220 个数据科学任务上分析了常见智能体框架和模型的失败模式，并构建了可复用测试套件，针对六类特定失败进行受控测试。
正文 · 原文
Abstract:Large language models have significantly lowered the barrier to writing code. Naturally, this has also led to the development of data science agents that promise automating the design and execution of data science from task formulation to model development. Existing evaluations of agentic data science (agentic DS) assess downstream predictive performance and code correctness, providing preliminary insight in agentic DS capabilities. We argue that these evaluations overlook the fundamental principles of data science: methodological data science- and domain knowledge. As such, only assessing downstream performance masks methodological failures such as data leakage and inappropriate validation, limiting the development of robust agentic DS systems. In this paper, we evaluate agentic DS beyond accuracy from three different angles: 1) a large-scale analysis of failure modes in common agentic DS frameworks and models across 220 data science tasks, 2) a reusable and extensible test suite with curated data science tasks that test for six specific failures in a controlled manner, and 3) a qualitative assessment of agentic DS pipelines against expert solutions on Kaggle competitions. We find that the majority of agentic solutions contain at least one failure mode despite the complexity of agentic harness or model capacity. Moreover, we identify a positive correlation between the occurrence of data leakage failures and predictive performance, confirming the necessity to evaluate agents beyond accuracy. Our qualitative analysis shows that DS agents generally lack problem solving creativity and domain knowledge, compared to competition-winning data scientists. We believe that principled failure mode analysis, as presented here, should be integral part of agentic DS evaluations, and that the insights we surface will be key in developing trustworthy agentic DS pipelines.
| Comments: | preprint |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.14115 [cs.AI] |
| (or arXiv:2610.14115v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14115 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xue Li [view email]
[v1]
Fri, 9 Oct 2026 08:05:09 UTC (1,187 KB)
来源：arXiv cs.AI · arxiv.org