arXiv cs.CL· Tianyun Zhong, Wei Wang, Jialin Li, Jianhong Tu, Liyao Li, Chuqiao Kuang, Tianshi Zheng, Songchi Zhou, Weiqi Zhai, Qile Jin, Jiajia Li, Jiasheng Zheng, Boxi Cao, Yaojie Lu, Hongyu Lin, Shiwei Ye, Bing Zhao, Hu Wei, Lin Qu, Xianpei Han·· 12 小时前AI 评分45
SCI-AGENT-EXAM：面向科研任务的智能体基准测试
SCI-AGENT-EXAM: Benchmarking Agents on Scientific Research Tasks
AI 导读
研究者推出 SCI-AGENT-EXAM，一个面向科研工作流的基准，含 11 个科学领域、7 类研究任务的 120 个专家精选任务，要求智能体同时产出代码脚本与报告文本。评测采用程序化验证加专家校准 LLM 评判的混合协议，10 个前沿 LLM 智能体中 GPT-6 Astra 总分最高达 80.9，但仅 26.0% 的运行超过 0.95 成功率阈值。
正文 · 原文
Authors:Tianyun Zhong, Wei Wang, Jialin Li, Jianhong Tu, Liyao Li, Chuqiao Kuang, Tianshi Zheng, Songchi Zhou, Weiqi Zhai, Qile Jin, Jiajia Li, Jiasheng Zheng, Boxi Cao, Yaojie Lu, Hongyu Lin, Shiwei Ye, Bing Zhao, Hu Wei, Lin Qu, Xianpei Han
Abstract:Large language model (LLM) agents are increasingly capable of supporting scientific workflows, but their capabilities across diverse research activities remain difficult to assess systematically. Existing benchmarks often focus on selected scientific capabilities or task types, limiting a comprehensive characterization of scientific agents' strengths and weaknesses in realistic research workflows. We introduce SCI-AGENT-EXAM, a workflow-oriented benchmark comprising 120 expert-curated tasks across 11 scientific fields and 7 research task types defined by the Scientific Workflow Task Taxonomy (SWTT). Each task requires both executable artifacts (e.g., code and scripts) and textual artifacts (e.g., reports and explanations). To evaluate the heterogeneous task outputs, we develop a hybrid evaluation protocol combining deterministic programmatic verification for executable artifacts with a calibrated, expert-validated LLM-based judge for textual artifacts. Beyond outcome assessment, we analyze progress and solution stability along execution trajectories to characterize how agents reach their final results. Experiments with 10 frontier LLM agents reveal substantial performance gaps: GPT-6 Astra achieves the highest overall score of 80.9, yet only 26.0% of its runs exceed the 0.95 success threshold. Further analyses uncover distinct capability profiles across scientific task types and substantial variation in execution efficiency and stability. Our code and data are available at this https URL.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.32441 [cs.CL] |
| (or arXiv:2610.32441v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32441 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tianyun Zhong [view email]
[v1]
Fri, 23 Oct 2026 10:06:55 UTC (1,917 KB)
来源：arXiv cs.CL · arxiv.org