StructLens：用最大生成树为语言模型提供结构化视角

View PDF HTML (experimental)

Abstract:Language exhibits inherent structures, a property that explains both language acquisition and language change. Given this characteristic, we expect language models to manifest their own internal structures as well. While interpretability research has investigated how models compute representations mechanistically through attention patterns and Sparse AutoEncoders, the organization of the resulting representations is overlooked. To address this gap, we introduce StructLens, a framework to analyze representations through a holistic structural view. StructLens constructs maximum spanning trees based on the semantic representations in residual streams, inspired by tree representation in dependency parsing, and provides summaries of token relationships in representation space. We analyze how contiguous tokens are also nearby in representation space and find that middle layers show the strongest local-span organization. Moreover, analysis of pre-training checkpoints reveals that smaller local units become detectable earlier in pre-training, and larger units later. Our findings demonstrate that StructLens provides insights into how models organize token representations across layers and training. Our code is available at this https URL.

Comments:	NeurIPS 2026
Subjects:	Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2603.03328 [cs.CL]
	(or arXiv:2603.03328v3 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2603.03328 arXiv-issued DOI via DataCite

Submission history

From: Haruki Sakajo [view email]
[v1] Tue, 10 Feb 2026 11:30:32 UTC (2,701 KB)
[v2] Mon, 18 May 2026 04:36:15 UTC (1,521 KB)
[v3] Wed, 21 Oct 2026 02:41:46 UTC (1,874 KB)