arXiv cs.CL· Maayan Sharon, Tom Hope·· 3 天前AI 评分25
RATIO：科学文献中跨类型构思操作的检索基准
RATIO: A Benchmark for Retrieval Across Typed Ideation Operations in Scientific Literature
AI 导读
研究者提出 RATIO 基准，将科学文献检索的相关性定义为三种"构思操作"：Address 检索解决既定问题的方法、Broaden 检索更通用的表述、Specify 检索具体实例。该基准基于数百万篇 CS 全文论文构建，通过扩展话语标记远程监督并结合 LLM 与人工审核完成。实验显示，针对操作分别微调可显著提升检索器，LLM 重排序也有帮助，但仍有较大提升空间；数据、代码、模型与标注均已开源。
正文 · 原文
Abstract:Retrieved scientific literature can serve as inspiration for both human and AI scientists. Inspiration can take different forms: prior work may directly suggest how to address a problem, or surface directions at different levels of abstraction---zooming out to a more general view or zooming in to a concrete realization. We introduce RATIO (Retrieval Across Typed Ideation Operations), a large-scale benchmark in which relevance is defined by three operations which we name ideation moves: Address retrieves potential approaches for stated problems, Broaden retrieves more general formulations, and Specify retrieves concrete instantiations. RATIO is constructed from millions of full-text scientific papers across CS literature via a general recipe that extends discourse-marker distant supervision---previously used only for classification---to corpus-scale retrieval, combined with extensive LLM and human vetting. Experiments show that operation-specific fine-tuning substantially boosts retrievers and LLM reranking helps further, but much room for improvement remains. Researchers blind to condition found the fine-tuned retriever's candidates more useful as inspirations. RATIO provides a scalable training and evaluation framework for retrieval components that support literature-grounded ideation, opening up new research avenues in scientific inspiration retrieval. We release full data, code, models, and annotations.
| Subjects: | Computation and Language (cs.CL); Information Retrieval (cs.IR) |
| Cite as: | arXiv:2608.27394 [cs.CL] |
| (or arXiv:2608.27394v4 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2608.27394 arXiv-issued DOI via DataCite |
Submission history
From: Maayan Sharon [view email]
[v1]
Thu, 27 Aug 2026 17:24:58 UTC (2,681 KB)
[v2]
Mon, 31 Aug 2026 15:28:22 UTC (2,683 KB)
[v3]
Tue, 1 Sep 2026 11:22:32 UTC (2,682 KB)
[v4]
Tue, 13 Oct 2026 19:23:07 UTC (2,598 KB)
来源：arXiv cs.CL · arxiv.org