arXiv cs.AI· Hui Zhu, Saleh Zare Zade, Rafi Ibn Sultan, Xiangyu Zhou, Dongxiao Zhu·· 10 小时前AI 评分34
超越 Sink Token:视觉语言模型中的 Sink 引导键编辑
Beyond the Sink Token: Sink-Guided Key Editing in Vision-Language Models
AI 导读
研究提出 SHRINK,一种无需训练的方法,通过在生成过程中部分削减可见键中与 sink 对齐的分量来调控注意力。在采样推理下,该方法在 LLaVA-OneVision-7B 和 Qwen2.5-VL-3B 上分别将平均准确率提升 5.1 和 2.7 个百分点,在 Qwen2.5-VL-7B 与 Gemma-3-4B 上增益较小。
正文 · 原文
Abstract:In vision-language models (VLMs), a few sink tokens attract a disproportionate share of attention. While previous studies have examined why these tokens attract attention, we investigate how the sink-key direction affects attention to image and question tokens. Across four probed VLMs, boosting the magnitude of positive sink-aligned components increases attention on average, whereas boosting negative components decreases it; these responses persist through later layers. This identifies the native sink key as a reference for controlling attention beyond its own position. Motivated by this finding, we propose SHRINK, a training-free method that partially reduces sink-aligned components across visible keys during generation. On nine benchmarks under sampled reasoning, it improves mean accuracy by 5.1 and 2.7 percentage points on LLaVA-OneVision-7B and Qwen2.5-VL-3B, with smaller gains on the other two main models, Qwen2.5-VL-7B and Gemma-3-4B. Beyond SHRINK, rescaling sink-aligned key components can replace the attention operations of two existing attention-steering methods, GIFT and ACG, with comparable accuracy on hallucination benchmarks.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.23516 [cs.CV] |
| (or arXiv:2611.23516v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23516 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Dongxiao Zhu [view email]
[v1]
Tue, 17 Nov 2026 21:07:08 UTC (683 KB)
来源:arXiv cs.AI · arxiv.org