arXiv cs.AI· Junyu Guo, Yuchen Fang, Shangding Gu, Costas Spanos, James Demmel, Javad Lavaei·· 14 小时前AI 评分52
研究揭示 LLM 上下文更新失败：旧值绑定与注意力漂移
When Context Changes: Understanding Update Failures in LLMs
AI 导读
论文提出 LLM 智能体在上下文同时保留新旧信息时可能用旧值回答，称为 stale binding，并发布 Controlled In-Context Memory（CICM）基准用于追踪对话和智能体日志中的更新信息。
正文 · 原文
Abstract:As preferences, goals, and facts change, LLM agents must use the current state while earlier versions remain in context. Yet they can answer with an old value of the same variable, a failure that we call stale binding. To study when models use outdated information and why, we introduce Controlled In-Context Memory (CICM), a benchmark for tracking and using updated information in conversations and agent logs. We observe that even frontier reasoning models can fail to recover the current state. We find that in open-source models probes can still recover the updated value when the model answers with an old one, pointing to a failure to select information that remains available. Component tests in Qwen and Pythia identify a mechanism for this selection failure: attention drift, where attention favors old values over the current one when producing an answer. We study a one-layer transformer to mathematically understand how this phenomenon happens: when attention scores are similar, several old values can together receive more attention than the current value. Guided by this explanation, we redirect attention toward the current value without further training. When the current value is requested directly, adjusting this intervention for each input corrects most old-value errors across various model families while preserving nearly all initially correct answers. Reliable context management therefore requires more than remembering updated information: models must use it to guide their answers.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.38866 [cs.AI] |
| (or arXiv:2609.38866v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2609.38866 arXiv-issued DOI via DataCite |
Submission history
From: Junyu Guo [view email]
[v1]
Wed, 30 Sep 2026 03:19:14 UTC (1,000 KB)
[v2]
Thu, 29 Oct 2026 01:42:25 UTC (995 KB)
来源：arXiv cs.AI · arxiv.org