arXiv cs.CL· Mathis Immertreu, Achim Schilling, Thomas Kinfe, Patrick Krauss·· 13 小时前AI 评分24
在自然语言上训练的神经后继表征中涌现词类与构式类结构
Word-Class and Construction-Like Structure Emerges in Neural Successor Representations Trained on Natural Language
AI 导读
在 WikiText-103 上训练的残差网络无需词性监督即可预测三个时间尺度的后继表征（SR）分布，最短尺度下的无监督聚类稳健还原名词、动词和形容词，簇间有向转移复现了熟悉的句法不对称性。在更细粒度及 13 个词性类别上，同一几何结构揭示出跨越类别边界的语义—功能分组，以及指向候选日期、度量与标题—名称构式的有向关系。词性一致性随预测尺度延长而下降，表明词类是更丰富预测几何中的粗略区域。
正文 · 原文
Abstract:Neural language models are typically trained on next-token prediction, although linguistic structure spans multiple temporal scales. Successor representations (SRs) make this horizon explicit by encoding discounted distributions over future states. Here, we ask whether such predictive representations can recover not only word classes, but also finer functional and construction-like structure from natural language. A residual network trained on WikiText-103 predicts SR distributions at three horizons without part-of-speech supervision. At the shortest horizon, unsupervised clustering robustly recovers nouns, verbs, and adjectives, while directed inter-cluster transitions reproduce familiar syntactic asymmetries. At finer resolutions and across 13 part-of-speech categories, the same geometry reveals semantic-functional groupings that cross category boundaries and directed relations tracing candidate date, measurement, and title-name constructions. Part-of-speech agreement declines as the predictive horizon lengthens. These results suggest that word classes are coarse regions within a richer predictive geometry in which categorical and construction-like linguistic structure emerge from future-word distributions.
| Subjects: | Computation and Language (cs.CL); Neurons and Cognition (q-bio.NC) |
| Cite as: | arXiv:2605.24585 [cs.CL] |
| (or arXiv:2605.24585v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.24585 arXiv-issued DOI via DataCite |
Submission history
From: Mathis Immertreu [view email]
[v1]
Sat, 23 May 2026 13:55:09 UTC (541 KB)
[v2]
Mon, 5 Oct 2026 08:06:32 UTC (423 KB)
[v3]
Wed, 21 Oct 2026 09:14:42 UTC (433 KB)
来源：arXiv cs.CL · arxiv.org