arXiv cs.CL· Behzad Shayegh, Mohamed Osama Ahmed, Fred Tung, Leo Feng·· 11 小时前AI 评分38
LLM 能否对检索信息保持审慎？八款模型上下文确定性遵从性评估
Can LLMs Take Retrieved Information with a Grain of Salt?
AI 导读
研究评估八款 LLM 的“上下文确定性遵从性”，发现它们看到不确定上下文后无法复现此前输出、对表达的确定性响应失真，并过度依赖复杂上下文。作者提出结合先验提醒、上下文确定性重校准与上下文简化的交互策略，在不修改模型权重的情况下将遵从错误平均降低 25%。该工作被 NeurIPS 2026 接收。
正文 · 原文
Abstract:Large language models have demonstrated impressive retrieval-augmented capabilities. However, a crucial area remains underexplored: their ability to appropriately adapt responses to the certainty of the retrieved information. It is a limitation with real consequences in high-stakes domains like medicine and finance. We evaluate eight LLMs on their context-certainty obedience, measuring how well they adjust responses to match expressed context certainty. Our analysis reveals systematic limitations: LLMs fail to reproduce their prior outputs after observing an uncertain context, respond to expressed certainties in a distorted way, and overrely on complex contexts. To address these, we propose an interaction strategy combining prior reminders (feeding the model's own context-free answer back into the prompt), context-certainty recalibration, and context simplification. This approach reduces obedience errors by 25% on average, without modifying model weights, demonstrating the efficacy of interaction design in enhancing LLM reliability. Our contributions include a principled evaluation metric, empirical insights into LLMs' uncertainty handling, and a portable strategy to improve context-certainty obedience across diverse LLMs.
| Comments: | Accepted by The Fortieth Annual Conference on Neural Information Processing Systems (NeurIPS 2026) |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.06919 [cs.CL] |
| (or arXiv:2605.06919v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.06919 arXiv-issued DOI via DataCite |
Submission history
From: Behzad Shayegh [view email]
[v1]
Thu, 7 May 2026 20:29:22 UTC (359 KB)
[v2]
Fri, 6 Nov 2026 17:37:04 UTC (472 KB)
来源：arXiv cs.CL · arxiv.org