arXiv cs.CL· Alexandre Quemy·· 11 小时前AI 评分18
研究揭示 token 表征的深度流动呈非线性，且不沿自身密度下降
The Depth Flow of Token Representations Is Nonlinear and Does Not Descend Its Own Density
AI 导读
研究将 Pythia-160M 与 Pythia-410M 的语料均值轨迹拟合为离散 Langevin 模型，发现二次漂移在两个模型每一层转换上都优于线性映射。均值轨迹的单步近似仿射（仅 3% 非仿射），单个状态则达 17%；漂移下降的是一个势函数而非自身对数密度，旋转分量占可解释漂移的 4% 至 45%。token 在全部十三层、两个语料和两个模型上均保持角排名与局部密度排名。
正文 · 原文
Abstract:A token's representation is carried through the network layer by layer. The whole vocabulary carried together forms a flow. We fit this flow's equation of motion as a discrete Langevin model over corpus-mean trajectories of Pythia-160M and Pythia-410M, and score the predicted steps on held-out tokens. Linear maps are often used as cheap surrogates for a layer. The flow they summarize is not linear: a quadratic drift beats the linear map at every transition of both models, and the Kramers-Moyal estimator agrees wherever its neighborhoods stay local. At full width, the step of a mean trajectory is nearly affine, 3% non-affine, while the step of an individual state is not, 17%. We then characterize the flow further. First, we show that it does not descend its own log-density, whatever the diffusion matrix. The drift instead descends a potential that is not the density. Second, the rotational component is not negligible, 4 to 45% of the explainable drift, and it is the antisymmetric part of the linear map. Finally, a token keeps its angular rank and its local-density rank across all thirteen layers, on two corpora and both models.
| Comments: | Accepted at the NeurIPS 2026 Workshop on Symmetry and Geometry in Neural Representations (NeurReps), Proceedings Track, to appear in PMLR. 9 pages plus appendix. Code and results: this https URL |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2608.29706 [cs.CL] |
| (or arXiv:2608.29706v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2608.29706 arXiv-issued DOI via DataCite |
Submission history
From: Alexandre Quemy [view email]
[v1]
Sun, 30 Aug 2026 10:22:00 UTC (651 KB)
[v2]
Fri, 30 Oct 2026 08:58:46 UTC (656 KB)
来源：arXiv cs.CL · arxiv.org