arXiv cs.CL· Zhongping Ji·· 10 天前AI 评分22
HySPE：通过辛对偶剪切实现的位置编码
HySPE: Positional Encoding via Symplectic Dual Shears
AI 导读
研究者提出 Hyperbolic Symplectic Positional Encoding（HySPE），用辛变换的双剪切阻尼对称组合替代 RoPE 的旋转参数化，并通过特征基对角化与分块坐标重基消除表示漂移，在 RTX 4090 上前向延迟与缓存 RoPE 持平（7.21 ms）。
正文 · 原文
Abstract:We introduce Hyperbolic Symplectic Positional Encoding (HySPE), grounding positional attention in non-compact symplectic transformations. While canonical Rotary Position Embedding (RoPE) parameterizes the compact, elliptic branch of $\Sp(2,\R)$ via rotations, HySPE operationalizes its hyperbolic branch via a damped symmetric composition of dual shears, yielding a conformally symplectic contraction with two spectral decay rates per channel pair. To eliminate the exponential representation drift inherent to naive absolute factorizations, we diagonalize the operator in its invariant eigenbasis and introduce blockwise coordinate rebasing with adaptive centered execution. This guarantees length-independent numerical bounds while matching cached RoPE forward latency (7.21\,ms on an RTX 4090). On TinyShakespeare, HySPE-UltraLong maintains an invariant perplexity of 4.810 up to $16\times$ zero-shot extrapolation ($L=4096$), whereas RoPE degrades to 131.198. Scaled to a 51M-parameter subword Transformer on WikiText-103 ($L_{\text{train}}=512$), HySPE closely matches RoPE in-domain while robustly extrapolating to length 8192, reducing tail perplexity by 83.9\% over RoPE. While these controlled experiments establish HySPE's extrapolation robustness and numerical stability, evaluating its scaling behavior on large-scale foundation models remains an important direction for future investigation.
| Comments: | 11 pages |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.10154 [cs.CL] |
| (or arXiv:2610.10154v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.10154 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhongping Ji [view email]
[v1]
Wed, 7 Oct 2026 14:33:35 UTC (87 KB)
来源：arXiv cs.CL · arxiv.org