arXiv cs.CL· Benjamin Wilcox, Dawei Gao, Pradeeban Kathiravelu, Douglas Causey, Kewei Sha, Yunhe Feng·· 10 天前AI 评分23
ArcticQA：面向 LLM 北极科学弃答的数据集与基准
Arctic Questions, Missing Answers: A Dataset and Benchmark for LLM Abstention in Arctic Science
AI 导读
研究者发布 ArcticQA 数据集与 ArcticAbstain 基准，用于评估 LLM 在北极科学选择题中的弃答能力。ArcticQA 含 194 道源自北极一手研究的问题，并自动核查答案支持与干扰项矛盾；ArcticAbstain 对比答案存在与缺失两种条件。
正文 · 原文
Abstract:Large language models (LLMs) should abstain from scientific multiple-choice questions when no option is valid, but frequent abstention alone does not demonstrate sensitivity to answer availability. We introduce ArcticQA, a dataset of 194 questions derived from primary Arctic research, with automated checks of answer support and distractor contradiction against source evidence. We further develop ArcticAbstain, a paired benchmark comparing answer-present and answer-absent conditions, with the correct answer replaced by a distractor in the latter and an explicit abstention option in both. We evaluate eight models from the Gemini, Claude, and ChatGPT families at high reasoning effort, with three trials per condition, yielding 9,312 recorded responses. Answer-present abstention rates range from 0.0% to 63.0%, whereas replacing the correct answer increases abstention by 5.05 percentage points on average. These findings highlight substantial baseline differences and the need to evaluate abstention frequency and responsiveness jointly. The dataset and benchmark are available at this https URL.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.09446 [cs.CL] |
| (or arXiv:2610.09446v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.09446 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yunhe Feng [view email]
[v1]
Wed, 7 Oct 2026 04:58:28 UTC (87 KB)
来源：arXiv cs.CL · arxiv.org