arXiv cs.CL· Hiroaki Yamagiwa, Yusuke Takase, Hidetoshi Shimodaira·· 12 小时前AI 评分18
用投影核测量注意力头权重子空间之间的亲和度
Measuring Affinity between Attention-Head Weight Subspaces via the Projection Kernel
AI 导读
研究提出用基于主夹角的投影核（PK）量化注意力头权重矩阵所张成子空间之间的相似度，在 GPT2-small、BLOOM-560M 和 Qwen2-0.5B-Instruct 上，PK 比 Composition Score 等既有指标更清晰地复现了 IOI 任务中已知的头间交互。
正文 · 原文
Abstract:Understanding relationships between attention heads is essential for interpreting the internal structure of Transformers, yet existing metrics do not capture this structure well. We focus on the subspaces spanned by attention-head weight matrices and quantify head-to-head relationships using the Projection Kernel (PK), a principal-angle-based measure of subspace similarity. Experiments on GPT2-small, BLOOM-560M, and Qwen2-0.5B-Instruct show that PK reproduces known head-to-head interactions on the IOI task more clearly than prior metrics such as the Composition Score. We further introduce a framework to quantify the informativeness of PK distributions by comparing them with a reference distribution derived from random orthogonal subspaces. As an application, we analyze a directed graph constructed from PK and show that, in GPT2-small, L4H7 acts as a hub by functioning as an Identity Head.
| Comments: | EMNLP 2026 Main. Appendix formatted in single-column layout |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2601.10266 [cs.CL] |
| (or arXiv:2601.10266v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2601.10266 arXiv-issued DOI via DataCite |
Submission history
From: Hiroaki Yamagiwa [view email]
[v1]
Thu, 15 Jan 2026 10:37:55 UTC (3,127 KB)
[v2]
Mon, 2 Nov 2026 09:21:23 UTC (7,882 KB)
来源：arXiv cs.CL · arxiv.org