arXiv cs.CL· Mohammed Saqlain·· 13 小时前AI 评分52
真实 PMID 却对应错误论文：生物医学论断验证智能体的 harness 设计
Real PMIDs, Wrong Papers: Harness Design for Biomedical Claim-Verification Agents
AI 导读
研究者在 SciFact 生物医学论断验证任务上，以实时 PubMed API 为唯一工具，对四种 harness 做受控消融。单次检索仅找到 6% 的 gold 论文，让模型自行写查询把 gold 召回率提高到三倍，最佳准确率为 62%；给定 gold 摘要后同一模型可达 79-86%，说明检索而非推理是主要瓶颈。
正文 · 原文
Abstract:Literature-searching language agents are becoming a standard component of scientific assistants for the life sciences, yet there is little controlled evidence on which parts of their harness matter. We run a controlled ablation on biomedical claim verification (SciFact) with the live PubMed API as the only tool. Holding the backbone and output format fixed, we compare four harnesses: closed-book answering, single-shot retrieval using the claim as the query, a ReAct-style agent that writes its own PubMed queries, and an oracle given the gold abstracts, each with two efficient frontier backbones. Naive single-shot retrieval finds the gold paper for only 6% of claims and turns overconfident errors into abstentions without improving accuracy. Letting the model formulate its own queries triples gold recall and yields the best verdicts (62% accuracy vs. 49-59% for the other realistic harnesses), although these differences are not significant at our sample size and agents use fewer than half of their search budget. With gold abstracts, the same models reach 79-86% accuracy, a significant 17-24 point gap showing that retrieval, not reasoning, is the main bottleneck. Finally, a citation audit shows that closed-book models cite PubMed IDs that almost always exist (327/335) but are almost never relevant: a cross-family LLM judge rates only 3 of 327 existing citations as relevant to the claim, versus 67% for citations produced by the search agent. Existence checks are therefore insufficient safeguards for scientific agents. We release code, prompts, and all cached tool calls at this https URL.
| Comments: | Accepted to the Agentic AI for Biological Discovery (AgenticLS) Workshop at NeurIPS 2026 |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Information Retrieval (cs.IR); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.29812 [cs.AI] |
| (or arXiv:2610.29812v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.29812 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mohammed Saqlain [view email]
[v1]
Wed, 21 Oct 2026 14:35:56 UTC (27 KB)
来源：arXiv cs.CL · arxiv.org