跳到正文
arXiv cs.AI· Hui Zhu, Saleh Zare Zade, Rafi Ibn Sultan, Xiangyu Zhou, Dongxiao Zhu·· 10 小时前AI 评分34

超越 Sink Token:视觉语言模型中的 Sink 引导键编辑

Beyond the Sink Token: Sink-Guided Key Editing in Vision-Language Models

AI 导读

研究提出 SHRINK,一种无需训练的方法,通过在生成过程中部分削减可见键中与 sink 对齐的分量来调控注意力。在采样推理下,该方法在 LLaVA-OneVision-7B 和 Qwen2.5-VL-3B 上分别将平均准确率提升 5.1 和 2.7 个百分点,在 Qwen2.5-VL-7B 与 Gemma-3-4B 上增益较小。

正文 · 原文

View PDF HTML (experimental)

Abstract:In vision-language models (VLMs), a few sink tokens attract a disproportionate share of attention. While previous studies have examined why these tokens attract attention, we investigate how the sink-key direction affects attention to image and question tokens. Across four probed VLMs, boosting the magnitude of positive sink-aligned components increases attention on average, whereas boosting negative components decreases it; these responses persist through later layers. This identifies the native sink key as a reference for controlling attention beyond its own position. Motivated by this finding, we propose SHRINK, a training-free method that partially reduces sink-aligned components across visible keys during generation. On nine benchmarks under sampled reasoning, it improves mean accuracy by 5.1 and 2.7 percentage points on LLaVA-OneVision-7B and Qwen2.5-VL-3B, with smaller gains on the other two main models, Qwen2.5-VL-7B and Gemma-3-4B. Beyond SHRINK, rescaling sink-aligned key components can replace the attention operations of two existing attention-steering methods, GIFT and ACG, with comparable accuracy on hallucination benchmarks.
Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.23516 [cs.CV]
  (or arXiv:2611.23516v1 [cs.CV] for this version)
  https://doi.org/10.48550/arXiv.2611.23516

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Dongxiao Zhu [view email]
[v1] Tue, 17 Nov 2026 21:07:08 UTC (683 KB)

来源:arXiv cs.AI · arxiv.org