arXiv cs.AI· Yi-Cheng Lai, Hen-Hsen Huang·· 10 小时前AI 评分34
FutureBench:评估 LLM 知识-上下文冲突的可再生框架
FutureBench: A Renewable Framework for Evaluating Knowledge-Context Conflict in LLMs
AI 导读
FutureBench 是一个用带时间戳问题、受控段落三元组和无上下文探针评估 LLM 知识-上下文冲突的可再生框架。在 3,629 个问题和六个模型上,将检查与段落优先结合的 Verify 提示词使五个模型在截止前事件上更易采纳错误答案,主要挤占了 Baseline 下原本正确的回答;截止后事件仅 o4-mini 采纳率上升 13.7 个百分点。
正文 · 原文
Abstract:Asking a model to check retrieved text can also tell it which source to trust. We introduce FutureBench, a renewable framework for evaluating knowledge-context conflict using timestamped questions, controlled passage triplets, and no-context probes. Across 3,629 questions and six models, Verify (a prompt combining checking with passage priority) raises false-answer adoption in all five models tested on pre-cutoff events, mainly by displacing answers that were correct under Baseline, which allows both passage evidence and model knowledge. On post-cutoff events, only o4-mini increases adoption (+13.7 percentage points). A four-prompt comparison on 902 post-cutoff questions finds that passage priority without checking raises adoption, while moving to Verify lowers it in all five models. Adversarial-Aware, which warns of misleading passages and prioritizes model knowledge, restores 82.5-98.9% of displaced pre-cutoff answers with under one point of change in true-passage accuracy. FutureBench supports repeated evaluation on fixed cohorts and extension to new events as models change.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.23601 [cs.CL] |
| (or arXiv:2611.23601v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23601 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yi-Cheng Lai [view email]
[v1]
Tue, 17 Nov 2026 22:55:12 UTC (205 KB)
来源:arXiv cs.AI · arxiv.org