arXiv cs.AI· Qi Cheng, Rongchao Dong, Shengyu Chen, Licheng Liu, Dan Lu, Zhengzhang Chen, Wei Cheng, Yiqun Xie, Haifeng Chen, Xiaowei Jia, Haoyu Wang·· 10 天前AI 评分29
ST-Bench：面向科研任务的多智能体系统生成空间-时间基准
ST-Bench: A Spatial-Temporal Benchmark for Multi-Agent System Generation on Scientific Research Tasks
AI 导读
研究团队推出 ST-Bench，用于检验 LLM 多智能体系统（MAS）在真实科学数据分析中是否优于单智能体。该基准包含 100 个源自水文学、农业和湿地甲烷研究的数据科学任务，扩展为 2,067 条查询并经领域专家验证。在 GPT-5 基座下评测五种 MAS 生成方法，十种配置中九种超过最廉价的单智能体基线，最强配置综合得分接近其三倍，但推理时间约为四倍；增益主要来自覆盖率而非指标质量。
正文 · 原文
Authors:Qi Cheng, Rongchao Dong, Shengyu Chen, Licheng Liu, Dan Lu, Zhengzhang Chen, Wei Cheng, Yiqun Xie, Haifeng Chen, Xiaowei Jia, Haoyu Wang
Abstract:The rapid progress of LLM-based multi-agent systems (MAS) has shown that they largely outperform single agents on coding, math, and QA tasks, where executable tests provide a binary success signal. Whether this advantage transfers to real scientific data analysis remains untested. We introduce ST-Bench, a benchmark designed to answer two questions: whether MAS outperform single agents on complex scientific data analysis tasks, and if so, by how much and at what additional cost. ST-Bench contains 100 data science tasks adapted from published Earth science studies across hydrology, agriculture, and wetland methane research, expanded into 2,067 queries grounded in additional published studies and validated by domain experts. Using ST-Bench, we evaluate five recent MAS generation methods under two training protocols, against single-agent baselines on the same GPT-5 backbone. Nine of the ten MAS configurations exceed the cheapest single-agent baseline, with the strongest reaching nearly three times its composite score. This gain is primarily attributable to coverage: trained workflows produce realistic numerical metrics on a larger fraction of queries, while the quality of those metrics, conditional on producing realistic output, is comparable to that of the single-agent baseline. The strongest configuration requires approximately four times the single-agent inference time, whereas a more economical workflow captures the majority of the benefit at less than twice the cost. MAS specialization confers measurable benefit on scientific data analysis, but the benefit is conditional rather than universal.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.07763 [cs.AI] |
| (or arXiv:2610.07763v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.07763 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Qi Cheng [view email]
[v1]
Tue, 6 Oct 2026 04:59:49 UTC (299 KB)
来源：arXiv cs.AI · arxiv.org