arXiv cs.CL· Nikita Koriagin, Nikita Balagansky, Daniil Gavrilov·· 11 小时前AI 评分31
SAE 训练数据的缩放定律：固定预算下激活缓冲区该多大
Scaling Laws for SAE Training Data
AI 导读
研究提出 SAE 训练数据的缩放定律：总训练预算固定时，激活缓冲区增大到质量平台期后，重复回放缓冲区与全程使用新激活效果几乎相当，继续增加唯一激活对重建和可解释性收益甚微。
正文 · 原文
Abstract:Activation storage is often a bottleneck in sparse autoencoder (SAE) training. We study how many unique activations an SAE needs when its total training budget is fixed. Quality improves rapidly as the buffer grows, then reaches a plateau where replaying the buffer is nearly as effective as using fresh activations throughout training. Beyond this point, additional unique activations bring little benefit to reconstruction or interpretability, so the buffer need not grow with the training budget. We describe this tradeoff between diversity and repetition with a scaling law for limited data. We validate it across dictionary sizes and token budgets, early, middle, and late layers of Llama-3.1, Qwen3 model sizes, and BatchTopK, TopK, and JumpReLU SAEs. Sweeps using mathematical text and Python code test the effect of changing the activation source. Evaluation covers explained variance (EV), reconstruction MSE, feature absorption, automated interpretability, and sparse probing. These results give a practical training rule: choose the smallest buffer that reaches the quality plateau and replay it for the remaining budget. In our experiments, this reduces activation storage by $8\text{--}64\times$ with negligible quality loss.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.07756 [cs.LG] |
| (or arXiv:2611.07756v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.07756 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Nikita Balagansky [view email]
[v1]
Thu, 5 Nov 2026 10:00:55 UTC (89 KB)
来源：arXiv cs.CL · arxiv.org