跳到正文
arXiv cs.CL· Saeed AlMarri (Khalifa University)·· 9 小时前AI 评分18

零维注意力拓扑用于语言可接受性分类:BERT 注意力图 H0 表示未显优势

Zero-Dimensional Attention Topology for Linguistic Acceptability

AI 导读

一项研究将 144 个对称化 BERT 注意力图的最小生成树合并值汇总为 1440 维零维持续同调(H0)表示,用于语言可接受性分类。在 CoLA 验证集上,该拓扑表示的域外 Matthews 相关系数为 0.438,注意力统计量为 0.508,BERT 为 0.545。分析显示该表示保留 token 数量信息并丢弃一类方向性注意力差异,未证明其相对更简单的注意力摘要具有预测优势。

正文 · 原文

View PDF HTML (experimental)

Abstract:Topological summaries of Transformer attention provide sentence representations for linguistic acceptability classification. We examine a zero-dimensional persistent-homology ($H_0$) representation that summarizes minimum-spanning-tree merge values from 144 symmetrized BERT attention graphs into 1,440 features. The retrospective evaluation uses 8,517 downstream training sentences and 521 in-domain and 497 out-of-domain CoLA validation sentences after data-integrity filtering. The recorded out-of-domain Matthews correlation coefficients are 0.438 for the topological representation and 0.508 for attention statistics, compared with 0.545 for BERT. Attention-feature results are retained aggregate measurements; BERT predictions are rescored and length controls are refitted from the accompanying records. Analytical results show that the representation retains token-count information and discards a class of directional attention differences. Source resampling of BERT predictions also demonstrates the sensitivity of uncertainty estimates to the sampling unit. The reported results do not demonstrate a predictive advantage for this fixed topological representation over simpler attention summaries. They identify representation properties and evaluation controls relevant to subsequent studies of attention topology.
Comments: 10 pages, 4 figures, 4 tables. Code and data are provided as ancillary files
Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
Cite as: arXiv:2611.22312 [cs.CL]
  (or arXiv:2611.22312v1 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2611.22312

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Saeed AlMarri [view email]
[v1] Tue, 17 Nov 2026 07:08:36 UTC (876 KB)

来源:arXiv cs.CL · arxiv.org