arXiv cs.CL· Peiyu Zhang, Heng Ping, Shinibali Bhattacharyya, Anwesan Pal, Nikos Kanakaris, Kevin Cheang, Shixuan Li, Wei Yang, Paul Bogdan·· 11 小时前AI 评分34
QGPM：用查询引导的 RoPE 调制缓解长上下文 LLM 的“中间信息丢失”
Query-Guided RoPE Modulation for Lost-in-the-Middle Mitigation in Long-Context LLMs
AI 导读
研究者提出免训练方法 QGPM（Query-Guided Positional Modulation），利用模型自身注意力头得到的查询相关性重塑 RoPE 位置，缓解长上下文 LLM 的“lost-in-the-middle”问题。
正文 · 原文
Abstract:Long-context language models often neglect information in the middle of the input, a phenomenon known as "lost-in-the-middle". Part of this gap may stem from the long-range decay of rotary position embedding (RoPE), which weakens the positional signal between distant tokens. Existing input compression deletes low-importance tokens, and attention reweighting acts only after RoPE has weakened this signal. RoPE-level methods modify positions directly but apply the same remapping to every query. To address these limitations, we propose QGPM (Query-Guided Positional Modulation), a training-free method that reshapes RoPE positions using query relevance from the model's own attention heads. QGPM builds a broad relevance map for lower layers and a sharp one for upper layers. Low-relevance tokens advance by less than one position on fast-rotating planes, pulling relevant evidence toward the query. Slow-rotating planes keep the original layout, with every token in place. Experiments on NoLiMa and RULER with multiple context lengths show that QGPM improves over base models and strong baselines, with the largest gains on retrieval tasks, and generalizes across multiple backbones. Our code is available at this https URL.
| Comments: | 13 pages, 3 figures, 7 tables. Code: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.09554 [cs.CL] |
| (or arXiv:2611.09554v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.09554 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Peiyu Zhang [view email]
[v1]
Fri, 6 Nov 2026 15:01:03 UTC (372 KB)
来源：arXiv cs.CL · arxiv.org