arXiv cs.AI· Shicheng Liu, Krista Opsahl-Ong, Andrew Drozdov, Julian Killingback, Arnav Singhvi, Ronak Pradeep, Michael Bendersky, James C. Dickerson, Monica S. Lam·· 12 小时前AI 评分38
AI 智能体如何自动复现并检验科学发现的分析稳健性
AI-Driven Reproducibility and Analytical Robustness in Scientific Discovery
AI 导读
研究团队构建 SEER-Repro 基准，收录 20 篇基于 SEER 数据库的已发表论文及复现所需数据，并用 LLM 智能体成功复现了 2,190 条原子论断中的 77.4%。配套的智能体分析流水线会识别每项发现背后的分析选择、提出可辩护的替代方案并重跑分析，在完全复现的发现中标记出 9.5% 存在潜在关键分析选择，供专家复核。
正文 · 原文
Abstract:Reproducible and analytically robust research findings are essential to scientific progress, but evaluating these properties typically requires costly manual effort. We investigate whether LLM agents can help automate reproduction and robustness analysis, using a medical case study grounded in the Surveillance, Epidemiology, and End Results (SEER) database, a widely used U.S. cancer registry underlying tens of thousands of published studies. First, we introduce SEER-Repro, a benchmark of 20 published SEER-based papers together with the data and resources required to reproduce their findings. Second, we create an AI agent that successfully reproduces 77.4% of 2,190 atomic claims, while providing executable baselines for further analysis. Third, we develop an agentic analysis pipeline to test the robustness of the medical findings by identifying analytical choices underlying each finding, proposing defensible alternatives, and rerunning the analysis to determine whether those choices affect the conclusion. Across published papers in SEER-Repro, our pipeline identifies potentially consequential analytical choices in 9.5% of fully reproduced findings and flags them for expert review. These results show how LLM agents can support scientific review by reproducing findings, systematically testing their robustness, and directing expert attention toward potentially consequential analytical choices.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.23845 [cs.AI] |
| (or arXiv:2610.23845v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.23845 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shicheng Liu [view email]
[v1]
Fri, 16 Oct 2026 17:57:55 UTC (153 KB)
来源：arXiv cs.AI · arxiv.org