arXiv cs.AI· Mingtao Zhang, Guoli Yang, Zhanxing Zhu, Mengzhu Wang, Xiaoying Bai·· 15 小时前AI 评分20
ELinFormer：用熵校准线性注意力高效建模多变量时间序列
Efficient Linear Attention for Multivariate Time Series Modeling via Entropy Calibration
AI 导读
研究者提出熵校准线性注意力模块并实例化为 ELinFormer 预测器，将注意力聚合分解为均匀分量与查询相关的修正项，用查询专属温度校准二阶 KL 代理。该代理经泰勒展开化简为分数能量除以温度平方，可由中心化 key 的共享乘积算出而无需构建完整成对分数矩阵。
正文 · 原文
Abstract:Attention mechanisms are effective for multivariate time series modeling because they adaptively aggregate information across timestamps and variables. Their quadratic cost, however, limits their use in multivariate and spatio-temporal systems. Linear attention reduces this cost by replacing nonlinear attention mappings with forms that support associative computation. Replacing exponential weighting may weaken the amplification of high-score positions and further bias weakly selective distributions toward uniform aggregation. We propose an entropy-calibrated linear attention module and instantiate it as the ELinFormer forecaster. Our operator decomposes aggregation into a uniform component and a query-dependent correction, whose magnitude is calibrated by a query-specific temperature. We derive a second-order KL proxy from centered query-key scores. The Taylor expansion reduces the proxy to the squared-score energy divided by the squared temperature. This score energy can be computed from shared products of the centered keys without forming the full pairwise score matrix, and the same products support the linear attention output. We design a smooth temperature rule to calibrate the KL proxy: it keeps low-energy residuals vanishing, approximately calibrates moderate-energy queries to a shared KL level, and allows the proxy to continue growing for high-energy queries. Experiments on five PEMS benchmarks show competitive forecasting accuracy. Controlled comparisons with vanilla attention and Linformer demonstrate a favorable accuracy--memory trade-off, while analysis on PEMS08 shows that the proposed rational temperature rule and query-specific calibration achieve lower MAE than a linear rule or shared-temperature alternatives. These results support entropy calibration as an effective mechanism for preserving selective deviations in linear attention.
| Comments: | 10-page main paper with a 6-page supplement, 4 figures in total. Version 2 revises the theoretical formulation and proofs, adds a high-energy interpretation and controlled NaLaFormer comparison, and expands reproducibility material, and the core method is unchanged. Code on this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2511.03190 [cs.LG] |
| (or arXiv:2511.03190v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2511.03190 arXiv-issued DOI via DataCite |
Submission history
From: Mingtao Zhang [view email]
[v1]
Wed, 5 Nov 2025 05:07:55 UTC (54 KB)
[v2]
Sat, 24 Oct 2026 16:22:36 UTC (377 KB)
来源：arXiv cs.AI · arxiv.org