arXiv cs.CL· Muhammad Zeeshan Karamat, Syed Talal Hassan, Christiana Chamon Garcia·· 12 小时前AI 评分31
视觉语言模型的安全几何随深度变化：逐层定位与双向拒答控制
Depth-Dependent Safety Geometry in Vision-Language Models: Layer-Wise Localization and Bidirectional Refusal Control
AI 导读
研究提出一种无需重训或改参数的轻量推理时框架，用于在冻结的视觉语言模型中定位并操控安全敏感表征。在 Qwen3-VL-4B 上，逐层激活差异分析显示安全几何随深度非均匀分布，早期视觉层沿学习到的安全方向做激活引导可实现强双向拒答控制，不安全拒绝率跨度达 6%-96%，而较深层对引导基本不敏感，呈现表征可分性与因果可控性的分离。
正文 · 原文
Abstract:Vision-Language Models (VLMs) are widely deployed, yet the internal mechanisms behind their safety behavior remain poorly understood. We present a lightweight inference-time framework for localizing and manipulating safety-sensitive representations in frozen VLMs, without retraining or modifying parameters. On Qwen3-VL-4B, layer-wise activation divergence analysis reveals non-uniform safety geometry across depth. Causal interventions show that early vision layers support strong bidirectional refusal control through activation steering along a learned safety direction, with unsafe rejection spanning 6%-96%, whereas later layers respond more to activation magnitude scaling and exhibit a substantial safety-selectivity trade-off. Deeper layers remain well separated along the safety direction yet are largely insensitive to steering, distinguishing representational separability from causal controllability. InternVL2.5-4B shows qualitatively similar depth-dependent activation geometry despite differences in layer location and magnitude. Our results expose a white-box, representation-level refusal-bypass attack surface in VLM vision encoders and motivate inference-time stabilization of safety-sensitive representations.
| Comments: | Accepted at the NeurIPS 2026 Workshop |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL); Cryptography and Security (cs.CR); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.32182 [cs.CV] |
| (or arXiv:2610.32182v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32182 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Muhammad Zeeshan Karamat [view email]
[v1]
Fri, 23 Oct 2026 05:28:49 UTC (64 KB)
来源：arXiv cs.CL · arxiv.org