arXiv cs.AI· Yukun Dong, Linjun Zhang, Tian Zheng·· 12 小时前AI 评分31
作者参与式智能体可复现性审计系统在机器学习会议落地
Author-in-the-loop Agentic Reproducibility Auditing for a Machine Learning Conference
AI 导读
研究者提出一种作者参与式（author-in-the-loop）协作审计流程，把作者声明作为 AI 智能体调查与评估的显式输入，通过结构化清单记录复现说明与局限，并在 GitHub 上用两阶段工作流连接智能体检查与作者澄清、修订和智能体复评。
正文 · 原文
Abstract:Reproducibility checks during peer review require a shared understanding of what authors' code is expected to reproduce. We present an author-in-the-loop collaborative audit that makes author declarations explicit inputs to an AI agent's investigation and assessment. A structured checklist records reproduction instructions and limitations; a two-stage workflow on GitHub connects agent checks to author clarification, revision, and agent reassessment. The agent separately assesses whether results have been reproduced and whether authors need to provide clarification or make repairs, while human reviewers approve requests and decide whether concerns are resolved. We deployed the system across 36 conference papers and 133 experimental tables and figures, raising 20 issues after human review. Further checks supported closing all the issues, although full reproduction remained unverified in some cases. We further compared repeated audits with complete and blank checklists to examine how author declarations shaped the agent's investigation and assessment. Our work contributes a collaborative workflow and agent design that use author knowledge to guide reproducibility auditing. The conference deployment provides empirical evidence that this approach can help identify and address reproducibility problems through collaboration among authors, agents, and reviewers.
| Comments: | 23 pages, 8 figures, 4 tables |
| Subjects: | Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC) |
| Cite as: | arXiv:2610.22465 [cs.AI] |
| (or arXiv:2610.22465v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.22465 arXiv-issued DOI via DataCite |
Submission history
From: Yukun Dong [view email]
[v1]
Thu, 15 Oct 2026 17:28:51 UTC (762 KB)
来源：arXiv cs.AI · arxiv.org