arXiv cs.CL· Yuxuan Liu·· 13 小时前AI 评分38
First Passage Attention：用路径极值替代路径求和的新型注意力机制
First Passage Attention
AI 导读
研究者提出 First Passage Attention（FPA），将遗忘门机制从沿 key-query 路径的单调累加改为路径极值，按学习到的标量场在区间内的最深下探幅度成比例降低 key 的 logit，从而选择性擦除中间上下文而保留更早的结构相关 token。
正文 · 原文
Abstract:Augmenting softmax attention with forget gates typically relies on summing decay contributions along the key--query path. Because such accumulation is monotonic, these mechanisms cannot selectively erase intermediate context while preserving older, structurally relevant tokens. We propose First Passage Attention (FPA), which shifts from path summation to path extrema by lowering a key's logit proportionally to the deepest dip of a learned scalar field below that key's level over the intervening span. FPA encompasses the Forgetting Transformer (FoX) as a special case, and when the field and levels follow a shared learned walk, the alive keys with unpenalised logits naturally form a stack. We show that a constant-dimension FPA head realises any subset of keys left alive by subsequent dips, whereas softmax heads with positional biases or token-level forget gates require a head dimension linear in the number of keys. In synthetic experiments, one-layer FPA heads converge to these target sets while comparable softmax and FoX baselines fail on every seed. Scaling to a 110M-parameter language model, FPA matches FoX in perplexity and uniquely achieves perfect passkey retrieval at $8\times$ the training length, even with its key--value cache compressed to about $25\%$.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.22714 [cs.CL] |
| (or arXiv:2610.22714v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.22714 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yuxuan Liu [view email]
[v1]
Thu, 15 Oct 2026 22:31:45 UTC (1,045 KB)
来源：arXiv cs.CL · arxiv.org