arXiv cs.CL· Mohammad Aflah Khan, Krishna P. Gummadi, Manish Gupta, Abhilasha Ravichander·· 2 天前AI 评分31
部分 RoPE 能否兼顾效率与收敛？研究揭示性能与训练动态
Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE
AI 导读
研究系统考察了部分 RoPE（仅对一部分隐藏维度施加旋转位置编码）对 Transformer 训练动态与收敛的影响，发现仅对约 10% 维度应用 RoPE 即可达到与完整 RoPE 相当的收敛效果，同时相比标准 RoPE 缓存最多节省 10× 内存。
正文 · 原文
Abstract:Rotary Positional Embedding (RoPE) is widely used in transformers to encode relative positional information, yet the effect of applying RoPE to only a subset of hidden dimensions remains underexplored. This design choice can substantially reduce memory usage, especially at long context lengths. We show that partial RoPE achieves up to 10$\times$ memory savings over standard RoPE cache while maintaining comparable final loss. We systematically study its impact on training dynamics and convergence across architectures, model sizes, sequence lengths, and datasets. Our results reveal that: (1) applying RoPE to only a small fraction of dimensions (around 10\%) matches the convergence of full RoPE; (2) these trends remain consistent across settings; and (3) NoPE (No Positional Encoding) can produce loss spikes, which are mitigated by minimal RoPE usage or QK-Norm application. Overall, our findings highlight partial RoPE as an effective trade-off between efficiency and stability.
| Comments: | Accepted to EMNLP 2026 Main Conference as a Short Paper |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2603.11611 [cs.LG] |
| (or arXiv:2603.11611v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2603.11611 arXiv-issued DOI via DataCite |
Submission history
From: Mohammad Aflah Khan [view email]
[v1]
Thu, 12 Mar 2026 07:05:13 UTC (3,864 KB)
[v2]
Thu, 15 Oct 2026 08:33:35 UTC (3,883 KB)
来源：arXiv cs.CL · arxiv.org