arXiv cs.CL· Xinyu Guo, Minglai Yang, Clayton T. Morrison, Eduardo Blanco, Mihai Surdeanu·· 12 小时前AI 评分29
TaxoSAE：用分类学逻辑重构稀疏自编码器
TaxoSAE: Rethinking Sparse Autoencoders through Taxonomic Logic
AI 导读
TaxoSAE 将 is-a（上下位）分类关系显式编码进 SAE 特征空间，用每层一个 SAE 的堆叠结构对应分类层级，并引入分类对齐损失。在两个事实问答分类体系上，抽象概念上位词读出提升 31.3 分，概念替换成功率提升 25.5 分，未见概念经上位词读出最高提升 37.9 分。抑制某概念时仅其自身答案受影响（准确率 0-2%，其他概念仍为 91-100%）。
正文 · 原文
Abstract:Recent Sparse Autoencoders (SAEs) decompose the representations of large language models (LLMs) into interpretable features, enabling precise readout and control of encoded knowledge. However, existing SAEs treat each feature as a standalone unit, whereas real-world knowledge is often taxonomically organized. As a result, readout carries no guarantee of taxonomic consistency, and control lacks the granularity to specify the scope of an edit. We propose TaxoSAE, which explicitly encodes taxonomical information, built on is-a (hypernym-hyponym) relations, into the SAE feature space. TaxoSAE mirrors a taxonomy with a stack of SAEs, one per taxonomy level, and introduces a taxonomic alignment loss that trains each concept to be active whenever one of its hyponyms is. On two factual QA taxonomies, one over biographical relations and one over animals, TaxoSAE enables (1) more accurate readout through the taxonomy with concentrated activations, e.g., +31.3 points in reading out the hypernyms of abstract concepts never queried in training; (2) more precise control, with +25.5 points in swap success, while suppressing a concept removes only its own answers (0-2% accuracy vs. 91-100% on other concepts); and, more importantly, (3) readout of unseen concepts through their hypernyms, with up to +37.9 points in hypernym readout accuracy.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.31894 [cs.CL] |
| (or arXiv:2610.31894v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.31894 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xinyu Guo [view email]
[v1]
Thu, 22 Oct 2026 20:45:23 UTC (777 KB)
来源：arXiv cs.CL · arxiv.org