arXiv cs.AI· Swathi Shree Narashiman, Aravind Ramana Venkatramanan, Nambi Seshadri·· 3 天前AI 评分20
语言模型 Logits 的几何结构：通用规律与学习到的偏离
The Geometry of Language Model Logits: Generic Structure and Learned Departures
AI 导读
自回归语言模型每一步预测产生的词表 logit 向量，其有序头部 logits 在不同预测上下文、架构和规模下均遵循 log-linear 关系 z_(r) ≈ μ - α log r，并在预训练中逐步显现。受控次序统计基线可解释这一几何形态，却无法解释斜率 α 的系统性偏移：训练后模型 α ≈ 1.4–1.7，远高于初始化基线 α ≈ 0.2，且 α 与预测性能相关。
正文 · 原文
Abstract:Autoregressive language models produce, at every prediction step, a vocabulary-sized logit vector whose structure determines the conditional next-token distribution. We empirically identify that the ordered vocabulary-head logits follow a remarkably consistent log-linear relation
\[ z_{(r)} \approx \mu - \alpha \log r. \] across prediction contexts, architectures, and scales, emerging progressively during pretraining. To our knowledge, this is the first systematic analysis of the logit geometry underlying the distributional assumptions used in the prior decoding work. While controlled order-statistic baselines spanning bounded, light-tailed, and heavier-tailed distributions can largely explain the observed log-linear geometry, they do not explain the systematic shift in the slope $\alpha$: trained language models exhibit $\alpha \approx 1.4$--$1.7$, sitting well above the initialization baseline ($\alpha \approx 0.2$) . Moreover, $\alpha$ correlates with predictive performance, suggesting that it captures learned structure beyond generic extreme-value effects.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.18520 [cs.CL] |
| (or arXiv:2610.18520v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.18520 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Swathi Shree Narashiman [view email]
[v1]
Tue, 13 Oct 2026 15:11:06 UTC (653 KB)
来源：arXiv cs.AI · arxiv.org