arXiv cs.AI· Youran Sun, Xingyu Ren, Chugang Yi, Kejia Zhang, Di Wu, Yingkang Cao, Allen Zhou, Jianping Ye, Suying Liu, Jiaxuan Guo, Jianda Du, Weicheng Ye, Kecheng Liu, Hefan Zhang, Haixin Wang, Jie Xiao, Eric Yang, Haizhao Yang·· 2 天前AI 评分42
Sci$\Pi$vot：科学 AutoResearch 智能体关键决策时刻的首席研究员判断基准
Sci$\Pi$vot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents
AI 导读
研究者构建了 Sci$\Pi$vot，一个基于人类纠正记录的 248 例科学判断基准，源自资深研究员六个月内在 26 个项目上对 LLM 智能体的反复干预。在提供项目记录的中等推理强度下，五个前沿模型中有四个低于随机水平，最高准确率 63.5%，低于新手的 73.4%；且各模型带项目记录的准确率反而低于不带记录时，重新考虑也未能稳定提升表现。
正文 · 原文
Authors:Youran Sun, Xingyu Ren, Chugang Yi, Kejia Zhang, Di Wu, Yingkang Cao, Allen Zhou, Jianping Ye, Suying Liu, Jiaxuan Guo, Jianda Du, Weicheng Ye, Kecheng Liu, Hefan Zhang, Haixin Wang, Jie Xiao, Eric Yang, Haizhao Yang
Abstract:Large language model (LLM) agents can keep scientific experiments running for months with little human supervision, but reliable autonomous research also requires scientific judgment. Evaluating scientific judgment is difficult because research decisions are open-ended and depend on project context. We construct Sci$\Pi$vot, a 248-case benchmark of scientific judgment based on human corrections. The benchmark draws on records from six months of research in which senior researchers ran LLM agents on 26 projects across six scientific fields and repeatedly intervened to correct the agents' research judgments. In each case, an agent chooses between the original LLM plan and the human-corrected alternative, using only the project record available before the correction as context. We compare LLM and human performance and examine the effects of reasoning effort, access to the project record, and reconsideration. At medium reasoning effort with the project record, models systematically prefer the original LLM plan over the human-corrected alternative. Four of five frontier models perform below chance on the selected cases. The highest accuracy in the project condition across the tested models and reasoning effort settings is 63.5%, compared with 73.4% for fresh researchers. At medium effort, every model achieves lower accuracy with the project record than without it, and reconsideration does not consistently improve overall accuracy. Even with a human-corrected alternative available, frontier models struggle with the selected research decisions, while Sci$\Pi$vot supports systematic evaluation of scientific judgment for LLM agents.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.21082 [cs.AI] |
| (or arXiv:2610.21082v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.21082 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hefan Zhang [view email]
[v1]
Thu, 15 Oct 2026 01:34:44 UTC (126 KB)
来源：arXiv cs.AI · arxiv.org