arXiv cs.AI· Nipun D. Pathirage, Kwan Srijomkwan·· 1 天前AI 评分38
研究：LLM 何时会超出静态内部记忆去搜索？GPT-5 mini 与 Grok 4 Fast 行为对比
Confident Fools or Careful Searchers? A Behavioural Study of When LLMs Search Beyond Their Static Internal Memory
AI 导读
一项新基准用 400 道选择题测试五款商用 LLM 的搜索决策，题目混合应直接作答的常识题与训练截止后新闻事实题。GPT-5 mini 和 Grok 4 Fast 对超 90% 的时效题触发搜索、仅 1% 的常识题搜索，而 Mistral Small 只搜索 20% 的时效题、常选择弃答。在无显式时间线索的时效题上，两款选择性模型的 NAS 从约 0.72 降至 0.57–0.58。
正文 · 原文
Abstract:Agentic LLM assistants must repeatedly decide whether to answer from memory or to call a tool such as web search. Unnecessary searches cost latency and money; missing ones yield confident but unsupported answers. Prior work mostly measures verbalised confidence or trains new adaptive-retrieval methods, and rarely audits the search decisions that deployed products actually make. We introduce a benchmark of 400 multiple-choice questions that combines widely known inside-knowledge (IK) questions, which should be answered directly, with outside-knowledge (OK) questions about facts from news published after the models' training cut-offs (to our knowledge), which should trigger search or, without tools, abstention. Each item offers an abstain option, and two action-level scores, NAS and KIS, reward searching on OK questions and restraint on IK questions. Across five commercial LLMs with and without their vendors' web search, GPT-5 mini and Grok 4 Fast search on over 90% of OK but only 1% of IK questions, whereas Mistral Small searches on just 20% of OK questions, often abstaining instead of using its tool; one product's search indicator proved unusable. Searching is partly triggered by explicit temporal cues: on uncued OK questions, the NAS of the two selective models falls from about 0.72 to 0.57--0.58. The scores thus measure the search triggering of specific products, not metacognition in a strict sense.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.41124 [cs.AI] |
| (or arXiv:2610.41124v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.41124 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Pathirage Deelaka Nipun [view email]
[v1]
Fri, 30 Oct 2026 01:28:38 UTC (1,321 KB)
来源：arXiv cs.AI · arxiv.org