arXiv cs.AI· Jinho Choi, Hyesu Lim, Jaegul Choo, Steffen Schneider·· 12 小时前AI 评分29
超越 SAE：ConSDL 用稀疏编码实现一致的概念发现
Beyond SAEs: Consistent Concept Discovery with Sparse Coding
AI 导读
研究者提出 Consistent Sparse Dictionary Learning（ConSDL），用逐样本稀疏编码替代标准线性编码器，以解决 SAE 学到的字典在不同训练轮次间不一致的问题。在 16 个合成设置中，ConSDL 实现近乎完美的字典与系数恢复；在四个视觉和语言设置上，其字典一致性和系数一致性分别比六个 SAE 基线平均值提升 17.6% 和 35.9%。
正文 · 原文
Abstract:Sparse autoencoders (SAEs) explain foundation model representations by decomposing activations into sparse combinations of learned dictionary atoms. However, the learned dictionaries can vary across training runs, limiting the reliability and reproducibility of SAE-based analyses. We propose Consistent Sparse Dictionary Learning (ConSDL), which replaces the standard linear encoder with per-sample sparse coding. This is motivated by our synthetic analysis showing that coefficients from a single linear encoder pass are suboptimal compared with those obtained by per-sample optimization, suggesting that sparse coding is a reliable mechanism for dictionary learning. Across 16 synthetic settings, ConSDL achieves near-perfect dictionary and coefficient recovery. These gains translate to state-of-the-art cross-seed consistency across four vision and language settings, where ConSDL improves dictionary and coefficient consistency by 17.6% and 35.9%, respectively, over the average of six SAE baselines.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.23801 [cs.AI] |
| (or arXiv:2610.23801v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.23801 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jinho Choi [view email]
[v1]
Fri, 16 Oct 2026 17:25:11 UTC (11,913 KB)
来源：arXiv cs.AI · arxiv.org