SATFormer：让 Transformer 学会何时使用早期值残差

View PDF HTML (experimental)

Abstract:Recent Transformer architectures expose later layers to representations computed in earlier layers, motivated by the observation that low-level features can become harder to recover as the residual stream is transformed through depth. One such method adds static value residuals: learned mixing coefficients that expose the first-layer value projection V_1 uniformly across tokens and heads. However, we theorize that the usefulness of $V_1$ is unlikely to be constant across tokens, heads, and contexts, and therefore treat early-representation reuse as a retrieval problem. Here, we introduce Selective Access Transformer (SATFormer), which preserves the first-layer value pathway while controlling access with a context-dependent gate. Across models from 130M to 1.3B parameters, SATFormer improves validation loss over the static value-residual and Transformer baselines at every evaluated scale, and improves average zero-shot accuracy over both at all four scales. Its strongest gains appear on retrieval-intensive benchmarks, where it improves over static value residuals by approximately 1.5 average points, while maintaining near baseline throughput and memory. Mechanistic analyses suggest sparse, depth-dependent, head-specific, and category-sensitive access patterns, supporting the interpretation that SATFormer learns selective reuse of early representations rather than uniform residual copying. Our code is available at this https URL.

Subjects:	Machine Learning (cs.LG); Computation and Language (cs.CL)
Cite as:	arXiv:2605.03953 [cs.LG]
	(or arXiv:2605.03953v3 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2605.03953 arXiv-issued DOI via DataCite

Submission history

From: Skye Gunasekaran [view email]
[v1] Tue, 5 May 2026 16:38:29 UTC (338 KB)
[v2] Wed, 6 May 2026 01:32:44 UTC (338 KB)
[v3] Mon, 26 Oct 2026 17:34:32 UTC (342 KB)