跳到正文
arXiv cs.AI· Yi-Cheng Lai, Hen-Hsen Huang·· 10 小时前AI 评分34

FutureBench:评估 LLM 知识-上下文冲突的可再生框架

FutureBench: A Renewable Framework for Evaluating Knowledge-Context Conflict in LLMs

AI 导读

FutureBench 是一个用带时间戳问题、受控段落三元组和无上下文探针评估 LLM 知识-上下文冲突的可再生框架。在 3,629 个问题和六个模型上,将检查与段落优先结合的 Verify 提示词使五个模型在截止前事件上更易采纳错误答案,主要挤占了 Baseline 下原本正确的回答;截止后事件仅 o4-mini 采纳率上升 13.7 个百分点。

正文 · 原文

View PDF HTML (experimental)

Abstract:Asking a model to check retrieved text can also tell it which source to trust. We introduce FutureBench, a renewable framework for evaluating knowledge-context conflict using timestamped questions, controlled passage triplets, and no-context probes. Across 3,629 questions and six models, Verify (a prompt combining checking with passage priority) raises false-answer adoption in all five models tested on pre-cutoff events, mainly by displacing answers that were correct under Baseline, which allows both passage evidence and model knowledge. On post-cutoff events, only o4-mini increases adoption (+13.7 percentage points). A four-prompt comparison on 902 post-cutoff questions finds that passage priority without checking raises adoption, while moving to Verify lowers it in all five models. Adversarial-Aware, which warns of misleading passages and prioritizes model knowledge, restores 82.5-98.9% of displaced pre-cutoff answers with under one point of change in true-passage accuracy. FutureBench supports repeated evaluation on fixed cohorts and extension to new events as models change.
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.23601 [cs.CL]
  (or arXiv:2611.23601v1 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2611.23601

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Yi-Cheng Lai [view email]
[v1] Tue, 17 Nov 2026 22:55:12 UTC (205 KB)

来源:arXiv cs.AI · arxiv.org