Control Reinforcement Learning：用稀疏自编码器特征实现 LLM 可解释的 Token 级引导

View PDF HTML (experimental)

Abstract:Sparse autoencoders (SAEs) decompose language model activations into interpretable features, but existing methods reveal only which features activate, not which change model outputs when amplified. We introduce Control Reinforcement Learning (CRL), which trains a policy to select SAE features for steering at each token, producing interpretable intervention logs: the learned policy identifies features that change model outputs when amplified. Adaptive Feature Masking encourages diverse feature discovery while preserving singlefeature interpretability. The framework yields new analysis capabilities: branch point tracking locates tokens where feature choice determines output correctness; critic trajectory analysis separates policy limitations from value estimation errors; layer-wise comparison reveals syntactic features in early layers and semantic features in later layers. On Gemma 2 2B across MMLU, BBQ, GSM8K, HarmBench, and XSTest, CRL achieves improvements while providing per-token intervention logs. These results establish learned feature steering as a mechanistic interpretability tool that complements static feature analysis with dynamic intervention probes

Comments:	Accepted at NeurIPS 2026. 31 pages, 24 figures, 12 tables
Subjects:	Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
ACM classes:	I.2.7; I.2.6
Cite as:	arXiv:2602.10437 [cs.LG]
	(or arXiv:2602.10437v4 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2602.10437 arXiv-issued DOI via DataCite

Submission history

From: Seonglae Cho [view email]
[v1] Wed, 11 Feb 2026 02:28:49 UTC (17,666 KB)
[v2] Thu, 12 Feb 2026 02:43:40 UTC (17,666 KB)
[v3] Sun, 3 May 2026 22:52:48 UTC (17,693 KB)
[v4] Thu, 12 Nov 2026 06:59:02 UTC (17,671 KB)