arXiv cs.CL· Jingcheng Niu, Xingdi Yuan, Tong Wang, Hamidreza Saghir, Amir H. Abdi·· 11 小时前AI 评分44
Llama See, Llama Do：LLM 语境裹挟与分心的机制分析
Llama See, Llama Do: A Mechanistic Perspective on Contextual Entrainment and Distraction in LLMs
AI 导读
研究者在多种语言模型和提示设置中发现“语境裹挟”现象：模型对提示中先前出现过的 token 赋予显著更高的 logits，即使是随机 token，说明该现象是独立于语义相关性的机制性效应。反事实提示的影响大于事实性提示，表明语义因素会调节裹挟强度。作者用基于可微掩码的方法定位出“裹挟注意力头”，将其输出置零后裹挟效应显著减弱，模型输出回归到无干扰上下文时的结果。
正文 · 原文
Abstract:We observe a novel phenomenon, contextual entrainment, across a wide range of language models (LMs) and prompt settings, providing a new mechanistic perspective on how LMs become distracted by ``irrelevant'' contextual information in the input prompt. Specifically, LMs assign significantly higher logits (or probabilities) to any tokens that have previously appeared in the context prompt, even for random tokens. This suggests that contextual entrainment is a mechanistic phenomenon, occurring independently of the relevance or semantic relation of the tokens to the question or the rest of the sentence. We find statistically significant evidence that the magnitude of contextual entrainment is influenced by semantic factors. Counterfactual prompts have a greater effect compared to factual ones, suggesting that while contextual entrainment is a mechanistic phenomenon, it is modulated by semantic factors.
We hypothesise that there is a circuit of attention heads -- the entrainment heads -- that corresponds to the contextual entrainment phenomenon. Using a novel entrainment head discovery method based on differentiable masking, we identify these heads across various settings. When we ``turn off'' these heads, i.e., set their outputs to zero, the effect of contextual entrainment is significantly attenuated, causing the model to generate output that capitulates to what it would produce if no distracting context were provided. Our discovery of contextual entrainment, along with our investigation into LM distraction via the entrainment heads, marks a key step towards the mechanistic analysis and mitigation of the distraction problem.
| Comments: | ACL 2025 (Outstanding Paper) |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2505.09338 [cs.CL] |
| (or arXiv:2505.09338v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2505.09338 arXiv-issued DOI via DataCite |
Submission history
From: Jingcheng Niu [view email]
[v1]
Wed, 14 May 2025 12:33:05 UTC (599 KB)
[v2]
Fri, 27 Jun 2025 11:15:26 UTC (599 KB)
[v3]
Wed, 4 Nov 2026 04:51:11 UTC (595 KB)
来源：arXiv cs.CL · arxiv.org