arXiv cs.CY· Hayoung Jung, Pedro Viana Diniz, Jos\'e Reinaldo Corr\^ea Roveda, Abner Fernandes da Silva, Haeun Jung, Enoch Tsai, Aleksandra Korolova, Manoel Horta Ribeiro·· 10 小时前AI 评分58
AI 智能体能否在健康领域合成科学结论？SciConBench 给出评测
Can AI Agents Synthesize Scientific Conclusions in Health?
AI 导读
研究团队提出 SciConBench，一个包含 9.21K 问题与系统综述专家结论的大规模实时基准，用于评测健康与医学领域的开放域科学结论合成能力。在清洁室评测环境 SciConHarness 下，8 个前沿模型与深度研究智能体中表现最好的事实 F1 仅为 0.337，且清洁室设置下的成绩普遍低于无约束评测，说明数据泄漏会高估模型的真实合成能力。
正文 · 原文
Abstract:Scientific AI agents increasingly retrieve evidence, reason across sources, and synthesize conclusions used in consequential decisions. Yet, their ability to do so in high-stakes domains such as health remains unclear. We introduce SciConBench, a large-scale live benchmark of 9.21K questions and expert-written conclusions from systematic reviews to evaluate open-domain scientific conclusion synthesis in health and medical domains. The benchmark draws on an expert-validated automated evaluation pipeline that decomposes conclusions into atomic facts and measures correctness and comprehensiveness via factual precision and recall. To mitigate data leakage, we further introduce SciConHarness, a clean-room evaluation harness that equips agents with controlled web interaction to ensure valid measurement. Evaluating 8 frontier models and deep research agents, we find that factual quality remains low: under clean-room settings, the best agent achieves only a factual F1 of 0.337. Our clean-room setting consistently reduces performance relative to unconstrained evaluation, suggesting that leakage inflates estimates of models' true synthesis capabilities. Finally, we audit consumer-facing agents (e.g., Google AI Overview, OpenEvidence) and find they frequently generate incomplete and sometimes contradictory conclusions, even when the ground-truth answer is available. Overall, our results show that reliable synthesis of scientific conclusions in health remains an open challenge, and that clean-room evaluation is essential for assessing open-domain AI agents.
| Comments: | To appear at NeurIPS 2026 (Evaluations & Datasets Track). Please cite the NeurIPS version when the proceedings are available |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computers and Society (cs.CY) |
| Cite as: | arXiv:2606.11337 [cs.AI] |
| (or arXiv:2606.11337v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2606.11337 arXiv-issued DOI via DataCite |
Submission history
From: Hayoung Jung [view email]
[v1]
Tue, 9 Jun 2026 18:16:04 UTC (8,391 KB)
[v2]
Tue, 10 Nov 2026 04:31:28 UTC (8,814 KB)
来源：arXiv cs.CY · arxiv.org