arXiv cs.AI· Zhaoyi Liu, Haichuan Zhang, Xiaochuan Gong, Yancheng Zhu, Bowei Tian, Wanghao Ye, Yufan Zhang, Ang Li·· 10 小时前AI 评分31
拆解视觉语言模型中的多模态旋转位置编码（MRoPE）
Disentangling Multimodal Rotary Positional Embedding in Vision-Language Models
AI 导读
研究拆解了视觉语言模型广泛采用的多模态旋转位置编码（MRoPE）中视觉内部空间结构与跨模态空间效应的作用。跨多个模型规模、MRoPE 变体和 benchmark 的受控实验显示，跨模态空间效应作为空间参考几乎无额外收益，甚至可能损害模型性能，而视觉内部空间编码对细粒度定位仍然有用。
正文 · 原文
Abstract:Vision-language models widely adopt Multimodal Rotary Position Embedding (MRoPE), an extension of Rotary Position Embedding (RoPE) that incorporates temporal, height, and width axes to encode the spatial structure of visual tokens. However, MRoPE also assigns height and width coordinates to text tokens that lack inherent spatial structure, introducing spatial positional effects into text-vision attention. Existing MRoPE variants explore different frequency allocations and coordinate layouts, with their designs suggesting different intuitions about these cross-modal effects: they may provide useful spatial references for associating language with visual regions, or introduce unnecessary positional preferences that hinder text-vision interactions. In this work, we disentangle intra-visual spatial structure from cross-modal spatial effects to investigate this question. We identify a constraint of shared coordinate layouts that prevents eliminating cross-modal spatial effects while preserving intra-visual spatial encoding, and develop controlled experiments to isolate their respective contributions. Our results across multiple model scales, MRoPE variants, and benchmarks show that cross-modal spatial effects provide little additional benefit as spatial references and can even impair model performance, while intra-visual spatial encoding remains useful for fine-grained grounding.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.13334 [cs.AI] |
| (or arXiv:2611.13334v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13334 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhaoyi Liu [view email]
[v1]
Mon, 9 Nov 2026 23:00:25 UTC (13,237 KB)
来源：arXiv cs.AI · arxiv.org