arXiv cs.CL· Abraham Yeung·· 11 小时前AI 评分31
激活缓存压缩能压到多小？为 SAE 训练校准压缩方案
How Small Can We Go? Calibrating Activation-Cache Compression for SAE Training
AI 导读
研究用冻结-SAE 测试与重训练测试评估激活缓存压缩对 SAE 特征的影响，发现重训练后保留特征比例没有清晰尺度：GPT-2 缓存上 fp16 舍入保留 60.6% 特征，而 fp32 最小步长改动一个百万分之一的值仅保留 70.5%，换随机种子保留 23.0%。
正文 · 原文
Abstract:To train a sparse autoencoder (SAE), you need a massive cache of model activations. We look into the effects that compression has on an SAE in two ways: we keep the original fp32-trained SAE and pass the compressed cache into the same SAE to see how many feature activations stay the same (the frozen-SAE test), and we retrain an SAE on the compressed cache and see how many of the original features still exist (the retraining test). We found that in the retraining test, there is no clear scale for retained features. On a GPT-2 cache, rounding everything to fp16 keeps 60.6% of features, but changing one value in a million by the smallest step fp32 allows (a change eight orders of magnitude smaller) keeps only 70.5%. Meanwhile, changing the random seed keeps 23.0%. For the frozen-SAE test, the frozen SAE's correlation with the original increases in order of how precise the codecs are, but plain $L_2$ distance gives nearly the same ordering. We also see the same frozen reading correspond to different percentages retained at different sites. This leaves the question of the right measurement. We build one by calibrating: retraining on identical bytes sets the ceiling, and a reseed sets the floor. We then measure the effect of each compressed codec against its own site's floor. With TopK SAEs on GPT-2 small and Pythia-1.4B at research-scale budgets, naïve per-token int4 loses more features than a reseed everywhere we tested, as do PCA and random projection on GPT-2. Standardizing each channel before quantizing brings int4 back above the floor in terms of retained features, at the same raw file size. At our smallest budget, a bf16 forward pass, which some pipelines use to save memory, loses slightly more features than rotated 8-bit storage. Calibrating a site costs four training runs and two evaluation passes.
| Comments: | Accepted at the NeurIPS 2026 Workshop on Interpretability as a Science (InterpScience), non-archival. 27 pages, 5 figures, 30 tables |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.08551 [cs.LG] |
| (or arXiv:2611.08551v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.08551 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Abraham Yeung [view email]
[v1]
Thu, 5 Nov 2026 10:09:19 UTC (668 KB)
来源：arXiv cs.CL · arxiv.org