arXiv cs.AI· Hunjae Lee, Corey Clark·· 12 小时前AI 评分22
动态关系预激活（prime attention）提升 Transformer 多变量时间序列预测能力
Dynamic Relational Priming Improves Transformer in Multivariate Time Series
AI 导读
研究者提出动态关系预激活注意力（prime attention），通过可学习调制为每个 token 对动态定制表示，替代标准注意力中跨所有配对计算不变的静态 token 表示，计算复杂度与标准注意力同阶。在多变量时间序列基准上，prime attention 持续优于标准注意力，预测精度最高提升 6.5%，并在序列长度减少最多 40% 时达到相当或更优性能。
正文 · 原文
Abstract:Standard attention mechanisms in transformers employ static token representations that remain unchanged across all pair-wise computations in each layer. This limits their representational alignment with the potentially diverse relational dynamics of each token-pair interaction. While they excel in domains with relatively homogeneous relationships, standard attention's static relational learning struggles to capture the diverse, heterogeneous inter-channel dependencies of multivariate time series (MTS) data--where different channel-pair interactions within a single system may be governed by entirely different physical laws or temporal dynamics. To better align the attention mechanism for such domain phenomena, we propose attention with dynamic relational priming (prime attention). Unlike standard attention where each token presents an identical representation across all of its pair-wise interactions, prime attention tailors each token dynamically (or per interaction) through learnable modulations to best capture the unique relational dynamics of each token pair, optimizing each pair-wise interaction for that specific relationship. This representational plasticity of prime attention enables effective extraction of relationship-specific information in MTS while maintaining the same asymptotic computational complexity as standard attention. Our results demonstrate that prime attention consistently outperforms standard attention across benchmarks, achieving up to 6.5\% improvement in forecasting accuracy. In addition, we find that prime attention achieves comparable or superior performance using up to 40\% less sequence length compared to standard attention, further demonstrating its superior relational modeling capabilities.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2509.12196 [cs.LG] |
| (or arXiv:2509.12196v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2509.12196 arXiv-issued DOI via DataCite |
|
| Journal reference: | Proceedings of the 43rd International Conference on Machine Learning (ICML). 2026 |
Submission history
From: Hunjae Lee [view email]
[v1]
Mon, 15 Sep 2025 17:56:15 UTC (1,606 KB)
[v2]
Sat, 23 May 2026 18:24:30 UTC (2,551 KB)
[v3]
Thu, 22 Oct 2026 19:21:07 UTC (2,549 KB)
来源：arXiv cs.AI · arxiv.org