ARE-bench：在真实研究者会话与工作区中评测智能体的科研能力
ARE-bench: Benchmarking Agents' Research Ability in Real-World Researcher Sessions and Workspaces
研究者提出 ARE-bench，用真实研究者项目中的智能体会话和完整工作区评测科研智能体能否推进进行中的项目。基准含 75 个任务、7 个研究阶段、33 项需求和 150 项子需求，共 1,173 条二元检查项。七个前沿模型中最强者仅 66.2% 的任务通过全部检查项；仅检查交付契约与通用规范时三个模型通过约 69%，加入历史与材料/工作区要求后降至 28.4% 至 40.0%。
Authors:Jiahang Lin, Wanying Ren, Wanghan Xu, Huayu Sha, Yang Shu, Zhonghang Lu, Chengjun Pan, Yi Dong, Shichun Liu, Yuyang Hu, Shihan Dou, Honglin Guo, Huijie Lv, Pujun Zheng, Zhe Sun, Yuankai Ying, Yiding Wang, Jiahao Chen, Zhixiong Yang, Dingbo Yuan, Yicheng Zou, Zhecan Wang, Guoxiu He, Xiaoyu Xing, Qi Zhang, Tao Gui, Xuanjing Huang
Abstract:Research agents achieve increasingly strong benchmark scores, yet these scores do not necessarily translate into reliable delivery in ongoing research projects. Many research-agent benchmarks emphasize standalone tasks and relatively self-contained workflows in clean or curated workspaces. However, researchers ask agents to complete one step within an ongoing project, where requests depend on earlier session decisions and workspaces contain unfinished files, outdated versions, and failed attempts. We introduce ARE-bench to evaluate whether research agents advance ongoing projects while respecting prior decisions and existing workspace state, grounded in real session context and in-progress workspaces. ARE-bench includes 75 tasks built on agent sessions and full workspaces from recruited researchers' own projects, spanning seven research stages, 33 requirements, and 150 sub-requirements, with 1,173 binary checklist items in total. Each task is anchored at a real user request, preserves its requirements, and summarizes the earlier requests still in effect as background. The complete workspace is replayed to that boundary without cleanup, preserving artifacts from earlier turns. Each checklist is compiled from an intent-aligned rubric and credits any valid solution. Across seven frontier models, the strongest passes every checklist item on only 66.2% of tasks, and Design and Execution rank among the hardest stages for every model. Three models pass about 69% of tasks when only delivery contracts and general norms are checked, but just 28.4% to 40.0% once historical and material or workspace requirements are also checked. Evaluating delivery without these contextual requirements can therefore overestimate research agents and mask differences between them.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.14364 [cs.CL] |
| (or arXiv:2611.14364v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.14364 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shichun Liu [view email]
[v1]
Tue, 10 Nov 2026 17:21:31 UTC (1,214 KB)
来源：arXiv cs.CL · arxiv.org