arXiv cs.AI· Yuchen Zhang, Taolin Han, Jinghang Wang, Jinxin Wang, Yun Wu, Wai Yuet Chiu, Xiaogang Li, Yuting Fu, Kewei Sun, Jiaxin Li, Qile Jin, Jiajia Li, Ruodan Chen, Qingteng Chen, Jialu Nie, Junchao Li, Yifei Zhang, Chen Zhao, Hu Wei, Lin Qu, Yanni Yin, Bing Zhao·· 4 小时前AI 评分38
SciWorkBench:面向真实科学工作流的智能体基准测试
SciWorkBench: A Benchmark for Real Scientific Workflows
AI 导读
研究者提出 SciWorkBench,一个覆盖生物学、化学、材料科学与物理学的真实科研任务基准,用于评估语言模型智能体能否完成科学家式项目。在 Claude Code 与 Codex 框架下统一评测,表现最好的智能体平均得分仅 57.38。失败分析指出三类工作流层面弱点:难以将科学知识转化为任务步骤、跨模态证据整合薄弱、科学自我纠错能力有限。
正文 · 原文
Authors:Yuchen Zhang, Taolin Han, Jinghang Wang, Jinxin Wang, Yun Wu, Wai Yuet Chiu, Xiaogang Li, Yuting Fu, Kewei Sun, Jiaxin Li, Qile Jin, Jiajia Li, Ruodan Chen, Qingteng Chen, Jialu Nie, Junchao Li, Yifei Zhang, Chen Zhao, Hu Wei, Lin Qu, Yanni Yin, Bing Zhao
Abstract:Language models are increasingly being used as agents capable of planning and executing projects, as well as refining their actions through feedback. This shift raises a question: can current agents complete projects like real scientists? To examine this question, we introduce SciWorkBench, a benchmark for evaluating agents' abilities on real science tasks across biology, chemistry, materials science, and physics. SciWorkBench is built from peer-reviewed literature or expert-contributed research scenarios. We also propose a unified scientific analysis framework validated by domain experts, which can categorize most tasks in SciWorkBench. Each task covers one or more stages of this framework. We evaluate up-to-date models within the Claude Code and Codex agent frameworks under a unified execution and grading protocol. The best-performing agent achieves a mean score of 57.38, indicating that current agents still struggle with realistic scientific workflows. Our failure analysis identifies three recurring weaknesses at the workflow level: difficulty translating scientific knowledge into task-specific procedures, weak integration of evidence across different modalities, and limited capacity for scientific self-correction. These findings highlight the importance of workflow-based evaluation for measuring progress toward reliable AI systems for scientific research.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.22490 [cs.AI] |
| (or arXiv:2611.22490v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.22490 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Taolin Han [view email]
[v1]
Tue, 17 Nov 2026 09:46:16 UTC (1,466 KB)
来源:arXiv cs.AI · arxiv.org