arXiv cs.CL· Marco Bronzini, Carlo Nicolini, Bruno Lepri, Jacopo Staiano, Andrea Passerini·· 13 小时前AI 评分32
Hyperdimensional Probe：用向量符号架构解码 LLM 表征
Hyperdimensional Probe: Decoding LLM Representations via Vector Symbolic Architectures
AI 导读
研究者提出 Hyperdimensional Probe，一种结合符号表示与神经探测的混合监督探针，借助 Vector Symbolic Architectures（VSAs）与超向量代数统一了监督探针、SAE 稀疏代理空间和 logit 输出分析三种路径。
正文 · 原文
Abstract:Despite their capabilities, Large Language Models (LLMs) remain opaque with limited understanding of their internal representations. Current interpretability methods either focus on input-oriented feature extraction, such as supervised probes and Sparse Autoencoders (SAEs), or on output distribution inspection, such as logit-oriented approaches. A full understanding of LLM vector spaces, however, requires integrating both perspectives, something existing approaches struggle with due to constraints on latent feature definitions. We introduce the Hyperdimensional Probe, a hybrid supervised probe that combines symbolic representations with neural probing. Leveraging Vector Symbolic Architectures (VSAs) and hypervector algebra, it unifies prior methods: the top-down interpretability of supervised probes, SAE's sparsity-driven proxy space, and output-oriented logit investigation. By combining the supervised learning paradigm of traditional probes with the dictionary-based representation principle of SAEs, our approach enables deeper input-focused feature extraction while supporting output-oriented analysis. Our experiments demonstrate that our approach consistently extracts meaningful semantic information across different LLMs, embedding sizes, and configurations, uncovering concept-oriented insights into LLM inference across two distinct scenarios: input-completion tasks and QA-focused text generation. VSA-based probing overcomes the limitations of logit-based analyses, which are constrained by the model's token vocabulary, while also mitigating the noisier interpretability outcomes often produced by SAEs in settings with a bounded conceptual feature space. By supporting a joint investigation of input-output features, this work advances the semantic understanding of neural representations while unifying the complementary perspectives of prior methods.
| Comments: | Published in Transactions on Machine Learning Research (TMLR) |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2509.25045 [cs.CL] |
| (or arXiv:2509.25045v4 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2509.25045 arXiv-issued DOI via DataCite |
|
| Journal reference: | Transactions on Machine Learning Research (TMLR), 2026 |
Submission history
From: Marco Bronzini [view email]
[v1]
Mon, 29 Sep 2025 16:59:07 UTC (3,200 KB)
[v2]
Tue, 2 Dec 2025 13:09:44 UTC (3,200 KB)
[v3]
Tue, 21 Jul 2026 17:53:19 UTC (3,261 KB)
[v4]
Wed, 21 Oct 2026 09:31:21 UTC (3,276 KB)
来源：arXiv cs.CL · arxiv.org