arXiv cs.CL· Zhangchi Feng, Harry Jiang, Yangyifei Luo, Ray Zhang, Richong Zhang·· 11 小时前AI 评分31
DAMI：面向门控线性注意力长上下文检索的双轴记忆初始化方法
DAMI: Dual-Axis Memory Initialization for Enhanced Long-Context Retrieval in Gated Linear Attention
AI 导读
研究者提出双轴记忆初始化方法 DAMI，在训练前联合配置门控线性注意力中的 A_log 与 dt_bias 参数，按头分配几何时间尺度、按层设置低-高-低输入敏感度，并用少量激活样本校准实现，不增加可训练参数。
正文 · 原文
Abstract:Linear attention scales linearly with sequence length, but its finite-state memory can limit long-context retrieval. In gated linear recurrences, memory retention depends on state decay, which is shaped jointly by gate parameters A_log and dt_bias and by input activations. In controlled Gated DeltaNet (GDN) experiments, both parameters remain highly correlated with their initial values after training, highlighting the importance of initialization. Therefore, we introduce Dual-Axis Memory Initialization (DAMI), which jointly configures these parameters before training. DAMI assigns geometric timescales across heads for varied retrieval distances and a low-high-low input-sensitivity profile across layers. A small activation sample calibrates the GDN-family implementation to match both behavioral targets without adding trainable parameters. Across GDN, KDA, GDN2, Mamba-2, and RWKV-7, DAMI improves average RULER-style retrieval accuracy by 5.29 to 16.71 percentage points over the corresponding random-initialization baselines. Performance on language modeling, commonsense reasoning, and 2K real-world retrieval remains broadly comparable to those baselines. Our code is available at this https URL.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.41994 [cs.AI] |
| (or arXiv:2610.41994v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.41994 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhangchi Feng [view email]
[v1]
Fri, 30 Oct 2026 16:39:37 UTC (576 KB)
来源：arXiv cs.CL · arxiv.org