arXiv cs.CL· Simin Huo, Ning Li·· 10 小时前AI 评分34
Periodic 与 Harmonic Periodic RoPE:面向 LLM 长度外推的位置编码方案
Periodic and Harmonic Periodic RoPE for Length Extrapolation in LLMs
AI 导读
研究者提出 Periodic RoPE(P-RoPE)与 Harmonic Periodic RoPE(HP-RoPE),配合滑动窗口注意力,让模型只需 W 个不同旋转角度即可缓解位置耗尽问题。
正文 · 原文
Abstract:The ability to process long contexts is crucial for large language models (LLMs) to perform long-horizon tasks. Rotary position embedding (RoPE) gives every token a new rotation angle, so a model eventually meets angles it never saw in training, i.e., position exhaustion. We propose Periodic RoPE (P-RoPE), a positional encoding mechanism that operates in conjunction with sliding window attention. It repeats the first sliding-window-size \(W\) angles of the RoPE table, so the model needs only \(W\) distinct angles. P-RoPE works but introduces a wrap error at the window boundary. We further propose Harmonic Periodic RoPE (HP-RoPE), which removes the error by choosing frequencies that complete whole turns in \(W\) tokens. We compare models that share the same architecture, data, and schedule and differ only in the window-layer position encoding: standard RoPE, P-RoPE, and HP-RoPE. The results show that P-RoPE and HP-RoPE keep perplexity nearly flat from the training length \(768\) to \(16\)k, while RoPE, NTK, and YaRN rise. In a format-free Passkey test at \(16\)k, they remain at \(99\)--\(100\%\), against \(11\)--\(37\%\) for those RoPE interpolations. The code is available at \href{this https URL}{this https URL}.
| Comments: | 5 pages |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.27980 [cs.CL] |
| (or arXiv:2605.27980v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.27980 arXiv-issued DOI via DataCite |
Submission history
From: Simin Huo [view email]
[v1]
Wed, 27 May 2026 05:18:00 UTC (110 KB)
[v2]
Thu, 19 Nov 2026 11:59:30 UTC (352 KB)
来源:arXiv cs.CL · arxiv.org