跳到正文
arXiv cs.AI· Shilong Zhao (Independent Researcher)·· 4 小时前AI 评分31

基于 Attention Residuals 的深度感知激活引导方法

Depth-Aware Activation Steering via Attention Residuals

AI 导读

研究者提出一种无需训练的深度感知激活引导流程,将 Attention Residuals 的可学习跨层通路与 Jacobian-lens 概念读出结合,在 Qwen3-0.6B 和 Qwen3.5-0.8B 上按深度权重分布 persona 向量,内部对齐与行为偏移均强于单层注入。

正文 · 原文

View PDF HTML (experimental)

Abstract:Activation steering is a simple, training-free way to modulate LLM behavior, yet existing methods typically inject vectors at a single layer and lack mechanisms for depth-aware propagation or internal verification. We combine two recent ideas - Attention Residuals (AttnRes) providing learnable cross-layer pathways, and Jacobian-lens (J-space) concept readout - into a lightweight depth-aware steering pipeline. On two small open models (Qwen3-0.6B and Qwen3.5-0.8B), distributing a persona vector according to AttnRes-style depth weights yields stronger internal alignment (deeper-layer concept cosine and rank, p<0.0001) and stronger behavioral shifts than single-layer injection. The effect is sign-specific: reverse vectors invert all six metrics, random directions degrade alignment (p<0.0001), and a short LoRA fine-tune erases the imprint. Injection rewires attention strictly downstream of the injection point (upstream JS = 0.00000). Three practical findings follow. First, a layer-wise geometric decomposition of cosine readouts: at the injection layer 123% of the increase is arithmetic, while 5-11 layers downstream 100% is dynamical. Second, an intensity window: effects are monotone over 0.3-0.7, then generation collapses (617 to 306 to 25 chars). Third, metric discipline: keyword-rule metrics fail LLM-judge validation (kappa <= 0.09), and two rule-based conclusions were retracted. Training the depth weights on a steering objective (200 steps, frozen backbone) raises final alignment 10x on held-out prompts. Cross-persona vectors retain 96-99% of the effect: specificity holds against random, not semantic, directions. We contribute a practical, low-overhead depth-aware steering method with a careful evaluation protocol. Code and data are released.
Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.21655 [cs.CV]
  (or arXiv:2611.21655v1 [cs.CV] for this version)
  https://doi.org/10.48550/arXiv.2611.21655

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Shilong Zhao [view email]
[v1] Tue, 27 Oct 2026 04:09:18 UTC (9 KB)

来源:arXiv cs.AI · arxiv.org