arXiv cs.AI· Delin Wang, Sinuo Wang, Rui Sun, Beidi Luan, Qiming Shi, Jing Li, Daxin Jiang, Zuo Bai·· 13 小时前AI 评分38
RubricAtlas：从专家报告中挖掘类型级评分标准，评估深度研究智能体
RubricAtlas: Mining Type-Level Rubrics from Expert Reports to Evaluate Deep Research Agents
AI 导读
RubricAtlas 从 3,079 份中文金融研究报告构建覆盖 99 种研究类型的类型级评分标准，用于评估深度研究智能体的分析完整性。实验显示单份报告仅覆盖其类型约一半的分析问题，选取不同报告的要求会改变评估分数与智能体排名；在十个智能体上，两条评审流水线达到 κ = 0.71 的检查点一致性和 Kendall τ = 0.87 的排名一致性。
正文 · 原文
Abstract:Deep research agents conduct multi-step Web searches and synthesise retrieved information into analytical reports. Evaluating these outputs requires checking which analyses they deliver, beyond the presence of relevant facts. Query-specific binary rubrics make individual requirements explicit, but task-level checklists may leave the overall analytical structure underexamined. We introduce RubricAtlas, which derives type-level analytical requirements from expert reports on different objects within the same research type and organises detailed checks around recurring analytical questions. The requirements give related tasks a shared standard for what to analyse, without prescribing their answers. Our framework links binary checkpoints to report passages and evaluates completeness separately from argument validity. Built from 3,079 Chinese financial research reports, RubricAtlas covers 99 research types. Reference-report selection experiments show that individual reports cover only about half of their type's analytical questions, and that selecting requirements from different reports changes evaluation scores and agent rankings. Across ten agents, two judge pipelines achieve checkpoint agreement of $\kappa$ = 0.71 and agent-ranking agreement of Kendall $\tau$ = 0.87. Controlled deletions reduce scores on the affected analyses, while padding produces no detectable increase. These findings support type-level rubrics for comparing the analytical completeness of Web deep research outputs, with individual decisions traceable to report evidence.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.36878 [cs.AI] |
| (or arXiv:2610.36878v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.36878 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zuo Bai [view email]
[v1]
Tue, 27 Oct 2026 09:18:00 UTC (188 KB)
来源：arXiv cs.AI · arxiv.org