arXiv cs.AI· Dayoon Ko, Jihyuk Kim, Soyeong Jeong, Young-jun Lee, Dahyun Lee, Juyeon Kim, Gunhee Kim, Moontae Lee, Kyungjae Lee·· 13 小时前AI 评分38
CompLit 基准：科学文献检索评测须覆盖隐性、累积与未满足需求
CompLit: Scientific Literature Search Benchmarks Must Cover Implicit, Cumulative, and Unmet Needs
AI 导读
研究者提出文献检索基准 CompLit，覆盖全部八个 arXiv 领域，并为隐性需求、累积需求、无文献可满足的未满足需求各设一个诊断场景。
正文 · 原文
Abstract:How much can we trust an agent to find the paper we need? As researchers, we search the literature from what we have read toward what we have not. We thus describe what we want in our own terms, which a paper may show only implicitly; our requirements grow as we read; and what we seek may lie in an empty region no paper yet covers. We introduce CompLit, a literature search benchmark spanning all eight arXiv domains, with one diagnostic setting for each of these needs. The strongest agents, Codex (GPT-5.6-Sol) and Claude Code (Opus 4.8), answer 86-87% of Standard queries, yet fall to 66-68% when requirements are implicit (implicit), lose another 6 and 15pp, respectively, as they accumulate (cumulative), and abstain on at most 53% of the queries no paper satisfies (unmet). No agent leads in every domain, and all are less accurate on papers from before 2019 or after 2023. Tracing their search, we find that agents often reach the right paper, or the closest one, yet misjudge it. Doing well on CompLit is a step toward agents that we can trust to find the paper we need, or to tell us that none exists.
| Subjects: | Digital Libraries (cs.DL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.28658 [cs.DL] |
| (or arXiv:2610.28658v1 [cs.DL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.28658 arXiv-issued DOI via DataCite |
Submission history
From: Dayoon Ko [view email]
[v1]
Sat, 10 Oct 2026 05:37:30 UTC (5,166 KB)
来源：arXiv cs.AI · arxiv.org