arXiv cs.AI· Stefan Krsteski, Charlotte Meyer, Guillaume Allegre, Tony O'Halloran, Alexandre Sallinen·· 10 小时前AI 评分42
Messier：面向跨基准智能体评估的高分辨率语料库
Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation
AI 导读
Messier 是一个统一语料库，收录 923,158 条记录，覆盖 30 个基准、745 个智能体、11,891 个任务和 71,969 个验证器，并将异构组件标准化为统一 schema。基于该语料库的分析显示，前沿进展在不同基准组间并不均衡：函数调用类评估已基本饱和，编程类提升最快，企业工作流仍最具挑战性。反事实重打分还表明，报告性能对验证器结果的聚合方式高度敏感。
正文 · 原文
Abstract:Comprehensively evaluating AI agents across interactive environments is difficult due to fragmented tasks, scaffolds, verifiers, and scoring rules. Unfortunately, existing efforts to unify these evaluations are limited in scale and domain, making costly reruns necessary and leaving available data incomparable. We introduce Messier, a unified corpus of 923,158 records spanning 30 benchmarks, 745 agents, 11,891 tasks, and 71,969 verifiers. Messier combines public evaluation results with new runs on six underrepresented professional and scientific benchmarks, standardizing their heterogeneous components into a common schema. Using this corpus, we show that frontier progress is uneven across benchmark groups, with function-calling evaluations largely saturated, programming improving fastest, and enterprise workflows remaining most challenging. Counterfactual rescoring further shows that reported performance can be highly sensitive to how verifier results are aggregated. Finally, we derive capability scores from our corpus that correlate with Epoch's Evaluation Capability Index rankings at Spearman \r{ho} = 0.84. The scores can also be estimated for subsets defined by domain, occupation, action space, or verifier type. In essence, Messier is a reusable resource for studying agent performance at scale, and a basis for designing better evaluations.
| Subjects: | Artificial Intelligence (cs.AI); Databases (cs.DB) |
| Cite as: | arXiv:2607.25891 [cs.AI] |
| (or arXiv:2607.25891v3 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.25891 arXiv-issued DOI via DataCite |
Submission history
From: Stefan Krsteski [view email]
[v1]
Tue, 28 Jul 2026 15:50:19 UTC (205 KB)
[v2]
Sun, 30 Aug 2026 18:22:18 UTC (202 KB)
[v3]
Tue, 10 Nov 2026 10:32:04 UTC (202 KB)
来源：arXiv cs.AI · arxiv.org