arXiv cs.AI· Naiyu Yin, Maosen Tang, Alex Townsend, Yue Yu·· 10 小时前AI 评分31
Rational Sparse Autoencoder:用可训练有理函数替代固定激活的稀疏自编码器
Rational Sparse Autoencoder
AI 导读
研究者提出 Rational Sparse Autoencoder(RSAE),用可训练有理函数替代 ReLU、JumpReLU、TopK 等固定编码器非线性,并证明在同等期望稀疏度下重建效果不劣于 ReLU。
正文 · 原文
Abstract:Sparse autoencoders (SAEs) are standard tools for mechanistic interpretability, but current SAE families are constrained by fixed encoder nonlinearities such as ReLU, JumpReLU, and TopK. This hard-codes a particular sparsity mechanism into the model and can distort the reconstruction-versus-sparsity trade-off. We introduce the Rational Sparse Autoencoder (RSAE), which replaces the fixed encoder activation with a trainable rational function. The design is motivated by a reconstruction advantage that we prove for RSAE at matched expected sparsity, with guaranteed non-inferiority to ReLU and, under stated assumptions, provable improvements over JumpReLU and TopK. The model is then implemented through a two-stage pipeline: an initialization procedure that copies the pre-trained baseline SAE weights and calibrates the scale parameters along with the rational coefficients; followed by a fine-tuning step under a sparsity-regularized reconstruction objective that holds $\ell_0$ at the teacher's value. Consistent with this theory, on residual-stream activations of three open-weight language models and across all three baseline activation families, the RSAE improves reconstruction fidelity after the fine-tuning step at the same level of sparsity. These gains are consistent across host language models, across baseline activation families, and across the full range of baseline sparsity we tested, while the upgrade itself adds only 20 scalar parameters per autoencoder and one training run takes between about 4 and 25 minutes on a single consumer GPU.
| Comments: | v2: M. Tang and A. Townsend added as co-authors; new theoretical analysis (Section 3, Appendix A); revised algorithm (Section 4, Appendix B.1); enhanced experiments (Section 5, Appendix B); v1 by N. Yin and Y. Yu |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2606.14990 [cs.LG] |
| (or arXiv:2606.14990v4 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2606.14990 arXiv-issued DOI via DataCite |
Submission history
From: Naiyu Yin [view email]
[v1]
Fri, 12 Jun 2026 22:22:40 UTC (116 KB)
[v2]
Tue, 16 Jun 2026 02:02:24 UTC (116 KB)
[v3]
Wed, 18 Nov 2026 01:34:25 UTC (105 KB)
[v4]
Thu, 19 Nov 2026 01:50:26 UTC (105 KB)
来源:arXiv cs.AI · arxiv.org