arXiv cs.CL· Li-Ming Yu, Hongyu Zhou·· 11 小时前AI 评分24
GatedV：面向 Transformer 语言模型的自门控 Value 注意力变体
GatedV: Self-Gated Values for Transformer Language Models
AI 导读
GatedV 是一种不增加可训练参数的注意力变体，仅在聚合前对 V 施加 SiLU。在 2.33M 参数下，其平均留出困惑度从 59.1 降至 57.9（2.0%），五个配对种子均改善；在 26.16M 参数下从 20.2 降至 19.7（2.4%），三个配对种子均改善。NVIDIA DGX Spark 上的短程交错基准显示，较大模型全上下文前向延迟约高 0.7%。
正文 · 原文
Abstract:Standard attention forms query (Q), key (K), and value (V) matrices through learned linear projections of input hidden states. Architectures such as GAU and Mega incorporate nonlinear transformations into attention. In this paper, we propose GatedV, a simple attention variant that applies SiLU only to V before aggregation without introducing additional trainable parameters. We evaluate GatedV against a standard decoder-only Transformer using matched parameter counts, paired initializations, and equal training-token budgets. At 2.33M parameters, GatedV reduces mean held-out perplexity from 59.1 to 57.9 (2.0%), with improvements across all five paired seeds. At 26.16M parameters, following an equal-budget learning-rate search, mean held-out perplexity falls from 20.2 to 19.7 (2.4%), with improvements across all three paired seeds. Small-model ablations show that the tested affine rescaling of values does not fully reproduce GatedV's gains. A short interleaved benchmark on NVIDIA DGX Spark measures approximately 0.7% higher full-context forward latency for the larger model, based on synchronized wall-clock timings. These findings suggest that value-only self-gating can improve language modeling performance without increasing parameter count or redesigning the attention-weighting mechanism.
| Comments: | 12 pages, 3 figures. Added Hongyu Zhou as a co-author. Code and experimental results are available at this https URL |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.02145 [cs.CL] |
| (or arXiv:2611.02145v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.02145 arXiv-issued DOI via DataCite |
Submission history
From: Li-Ming Yu [view email]
[v1]
Sun, 1 Nov 2026 19:24:49 UTC (183 KB)
[v2]
Tue, 3 Nov 2026 04:33:30 UTC (183 KB)
来源：arXiv cs.CL · arxiv.org