arXiv cs.CL· Heng Zhao, Yufei Wang, Ruidian Song, Weisheng Chen·· 12 小时前AI 评分31
高维层状态轨迹揭示 Transformer 推理的内部动态
High-dimensional layer-state trajectories reveal the internal dynamics of transformer inference
AI 导读
研究将 Transformer 推理中按层顺序展开的层状态轨迹作为可测量的计算过程，在 Qwen、Llama 和 Gemma 检查点上，真实层序路径与端点弦的对齐度是保持端点随机打乱的 1.60–2.24 倍。
正文 · 原文
Abstract:Mechanistic studies of large language models localize function in parameters, attention heads or neurons, but those parts do not specify how one inference unfolds. We treat the ordered layer-state trajectory as the empirical process and test whether layer order defines a measurable computation. Across representative Qwen, Llama and Gemma checkpoints, realized order produced paths 1.60--2.24 times as aligned with their endpoint chord as endpoint-preserving shuffles. The order effect transferred to a lexical task, but was stronger in three random initializations of the same Qwen architecture, identifying an architecture-compatible scaffold rather than a training-specific law. Direct hidden states provided stronger mechanism coordinates than vocabulary readbacks, and a task-defined coordinate made candidate competition computable. After additive interventions failed, local transition actuators selectively controlled internal trajectories across checkpoints. A confidence gate then drove 17 of 87 held-out Qwen closure trajectories across the full-vocabulary top-1 candidate boundary while leaving all 203 non-closure top-1 outputs unchanged. Safety-matched action reassignment performed similarly, locating the gain in selective gating and the bounded actuator family rather than fine-grained dose assignment. Layer order exposes an observable, computable and behaviourally consequential process of transformer inference without establishing a complete theory of LLM dynamics or control of answer correctness.
| Comments: | Expanded preprint with 17 main figures and 13 supplementary figures. Source data and code are archived on Zenodo |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.26866 [cs.LG] |
| (or arXiv:2610.26866v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.26866 arXiv-issued DOI via DataCite |
Submission history
From: Heng Zhao Edward [view email]
[v1]
Wed, 15 Jul 2026 00:02:19 UTC (1,695 KB)
来源：arXiv cs.CL · arxiv.org