arXiv cs.CL· William Timkey, Tal Linzen·· 3 天前AI 评分22
句法注意力头是否影响模型行为？
Do Syntax Heads Influence Model Behavior?
AI 导读
研究发现，即便强制 Transformer 语言模型中专门处理句法关系的注意力头去关注干扰名词，模型的句法预测仍更符合主语而非干扰项。对句法专用头的干预在两项指标中仅有一项显著强于随机头干预，说明这些头对模型句法预测的因果作用证据有限。
正文 · 原文
Abstract:Transformer language models develop attention heads that specialize in syntactic relations, for example, attending from verbs to their subjects. Do these heads play a causal role in the model's syntactic predictions? Evidence for such a role is mixed to negative: even when we force these heads to attend to a distractor noun other than the subject, the model's syntactic predictions are still more consistent with the subject than with the distractor. Interventions on specialized syntax heads had a significantly stronger effect on syntactic predictions than interventions on random heads, but only in one of the two measures we propose.
| Comments: | 4 pages, 2 figures |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.19231 [cs.CL] |
| (or arXiv:2610.19231v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19231 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: William Timkey [view email]
[v1]
Tue, 13 Oct 2026 21:47:26 UTC (116 KB)
来源：arXiv cs.CL · arxiv.org