语言模型能控制自己的注意力：Declarative Attention 让模型在思维链中声明关注范围

View PDF HTML (experimental)

Abstract:Language models spend most of their attention on a small fraction of context, yet they read the entire KV cache to find the few tokens that matter. If the user asks about a previous detail in a 1M-token conversation, global attention layers must scan the full context to generate each token of the reply. A prominent approach pre-selects relevant tokens via lightweight proxy scores, but this extrinsic scoring still incurs O(N) per step. We take an intrinsic approach motivated by the simple question: wouldn't the model already know which parts of the context are relevant? We introduce Declarative Attention (DA), a protocol that elicits the model to declare where it needs to attend within its chain-of-thought, partitioning generation into three modes: <global> (full context), <focus> (a specific region), and <local> (recent output only). The inference engine parses these declarations like tool calls and skips most of the KV cache read. Under zero-shot evaluation across 15 long-context tasks, DA on off-the-shelf models (Gemma-4-31B, Qwen-3.6-27B) reduces total attended tokens per response (45.8%, 19.6%), despite extra decoding steps, with modest accuracy drops (1.40pp, 2.10pp) that shrink with model scale. DA unlocks a new axis of sparse attention, with further potential under training-based methods. Code is available at this https URL.

Subjects:	Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Cite as:	arXiv:2609.02737 [cs.CL]
	(or arXiv:2609.02737v2 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2609.02737 arXiv-issued DOI via DataCite

Submission history

From: Namgyu Ho [view email]
[v1] Wed, 2 Sep 2026 15:43:38 UTC (314 KB)
[v2] Wed, 14 Oct 2026 23:22:42 UTC (359 KB)