arXiv cs.CL· Agnibh Dasgupta, Abdullah Tanvir, Xin Zhong·· 12 小时前AI 评分22
语言模型中的不变特征：几何刻画与模型归因
Invariant Features in Language Models: Geometric Characterization and Model Attribution
AI 导读
研究提出局部几何框架，将语义等价输入的表示刻画为潜空间中沿冗余方向变化、语义身份保存在不变子空间的结构，并给出不变潜特征的几何刻画与区分语义变化/语义保持变化的对比子空间发现方法。该不变表示可用于零样本模型归因，实验显示不变结构出现在特定深度区域，表示级干预表明不变成分对模型输出具有因果作用。论文已被 NeurIPS 2026 接收，代码已公开。
正文 · 原文
Abstract:Language models exhibit strong robustness to paraphrasing, suggesting that semantic information may be encoded through stable internal representations, yet the structure and origin of such invariance remain unclear. We propose a local geometric framework in which semantically equivalent inputs produce representations that occupy structured regions in latent space, with paraphrastic variation along nuisance directions and semantic identity preserved in invariant subspaces. Building on this view, we make three contributions: (1) a geometric characterization of invariant latent features, (2) a contrastive subspace discovery method that separates semantic-changing from semantic-preserving variation, and (3) an application of invariant representations to zero-shot model attribution. Across models and layers, empirical results support these contributions. Invariant structure emerges in specific depth regions, semantic displacement lies largely outside the nuisance subspace, and representation-level interventions indicate a causal role of invariant components in model outputs. Invariant representations also capture model-specific geometric patterns, enabling accurate attribution. Together, these findings support a local geometric characterization of semantic invariance in language model representations.
| Comments: | This paper is accepted to NeurIPS 2026. Code is available at this https URL |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.06458 [cs.LG] |
| (or arXiv:2605.06458v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.06458 arXiv-issued DOI via DataCite |
Submission history
From: Xin Zhong [view email]
[v1]
Thu, 7 May 2026 15:50:31 UTC (3,999 KB)
[v2]
Wed, 4 Nov 2026 20:46:10 UTC (4,006 KB)
来源：arXiv cs.CL · arxiv.org