arXiv cs.AI· Andrei-Tiberiu Pantea, Aymeric Dieuleveut·· 14 小时前AI 评分18
通过单点回归理解 TSFM 中的轴向注意力机制
Understanding axial attention in TSFMs through single location regression
AI 导读
研究将 Single Location Regression 框架扩展到 2D 结构化数据，用于分析时间序列基础模型（TSFM）中轴向注意力的理论性质。作者证明在特定渐近条件下，该轴向注意力型预测器可达到 Bayes 最优，而一类半线性简化模型严格无法做到，凸显内部非线性的作用。实验还表明逆温度调度对实际收敛至关重要。
正文 · 原文
Abstract:Recent advances in Time Series Foundation Models (TSFMs) increasingly rely on axial attention mechanisms, but despite their empirical success, the theoretical properties of axial attention remain largely unexplored. To bridge this gap, we extend the Single Location Regression framework introduced in Marion et al. [2025] to 2D structured data, in order to study sparse information settings where a target depends on a small number of tokens. We formulate an analytically tractable axial attention-type predictor and rigorously analyze its statistical properties and training dynamics. We prove that under a specific asymptotic regime and with knowledge of the underlying oracle parameters, our predictor achieves Bayes optimality, whereas a class of semi-linear simplifications strictly fails to do so, underlining the role of the inner non-linearity. We prove that for a specific temperature scaling, Projected Gradient Descent converges globally to the optimal oracle parameters. Extensive numerical experiments validate our theoretical findings and highlight the critical role of inverse temperature scheduling in empirical convergence.
| Subjects: | Machine Learning (stat.ML); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.40188 [stat.ML] |
| (or arXiv:2610.40188v1 [stat.ML] for this version) | |
| https://doi.org/10.48550/arXiv.2610.40188 arXiv-issued DOI via DataCite |
Submission history
From: Andrei-Tiberiu Pantea [view email]
[v1]
Thu, 29 Oct 2026 14:46:50 UTC (19,724 KB)
来源：arXiv cs.AI · arxiv.org