arXiv cs.CL· Prasoon Dev, Anirudh Sankar, Vasudeva Varma·· 6 天前AI 评分22
MS-GLA：用多时间分辨率解决 Gated Linear Attention 表征瓶颈
MS-GLA: Multi-Scale Gated Linear Attention for Addressing Representational Bottlenecks via Multi-Temporal Resolution
AI 导读
研究者提出多尺度门控线性注意力 MS-GLA，将注意力头分配到多个时间分辨率上，粗粒度头组池化更长 token 跨度以捕捉长程依赖，细粒度头组保留局部句法敏感度，并通过可学习的输入依赖融合层动态重组各头组输出。在参数量匹配的情况下，MS-GLA 在语言建模、召回密集任务和长上下文泛化上均优于 GLA，召回密集任务最高提升 18.9%，语言建模基准平均困惑度降低 9.5%。
正文 · 原文
Abstract:Gated Linear Attention (GLA) Transformers advance linear recurrent models through data-dependent gating, but face a core limitation: the fixed-capacity memory matrices across all heads operate at a single temporal resolution, where each token is processed individually, forcing them to simultaneously encode local syntactic patterns and long-range semantic structure, creating a representational bottleneck that gating alone is insufficient to resolve. We introduce Multi-Scale Gated Linear Attention (MS-GLA), which addresses this by distributing attention heads across multiple temporal resolutions. Coarser resolutions pool longer token spans naturally specializing toward long-range dependencies, while finer head groups retain sensitivity to local syntactic structure. A learnable, input-dependent fusion layer dynamically recombines head group outputs at each timestep, expanding effective memory capacity without increasing per-head state size. This multi-resolution decomposition draws on principles from Multi-Scale State-Space Models (MS-SSM), adapting them to the gated linear attention setting. We evaluate MS-GLA on language modeling, recall-intensive tasks, and long-context generalization. Across all settings, MS-GLA consistently achieves higher accuracy and lower perplexity than GLA at matched parameter counts, with up to 18.9% improvement on recall-intensive tasks and 9.5% lower average perplexity on language modeling benchmarks, validating multi-temporal resolution decomposition as a principled and effective extension of Gated Linear Attention.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.35664 [cs.CL] |
| (or arXiv:2609.35664v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2609.35664 arXiv-issued DOI via DataCite |
Submission history
From: Prasoon Dev [view email]
[v1]
Mon, 28 Sep 2026 17:23:45 UTC (572 KB)
[v2]
Fri, 9 Oct 2026 00:44:36 UTC (572 KB)
来源：arXiv cs.CL · arxiv.org