ACG：面向 LVLM 高效幻觉缓解的注意力空间对比引导

View PDF HTML (experimental)

Abstract:Hallucinations in large vision--language models (LVLMs) often arise when language priors dominate over visual evidence, leading to object misidentification and visually inconsistent descriptions. We address this problem by framing hallucination mitigation as contrastive guidance that steers generation toward visually grounded and semantically faithful text. We propose Attention-space Contrastive Guidance (ACG), a training-free, single-pass method that operates directly in self-attention layers, where hallucination-inducing cross-modal biases emerge. ACG constructs both image-conditioned and approximate text-only attention paths within a single forward pass, enabling efficient guidance before errors accumulate at the output layer. Because this masking-based surrogate can introduce approximation bias, we further apply a lightweight orthogonal projection that suppresses components aligned with the text-only path, yielding a more visually grounded correction. Experiments on CHAIR and POPE show that ACG improves faithfulness over existing training-free baselines while maintaining caption quality, reducing latency by up to $2\times$ compared to multi-pass contrastive decoding methods.

Comments:	Accepted at CVPR 2026 Findings. Code: this https URL
Subjects:	Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Cite as:	arXiv:2601.13707 [cs.CV]
	(or arXiv:2601.13707v3 [cs.CV] for this version)
	https://doi.org/10.48550/arXiv.2601.13707 arXiv-issued DOI via DataCite

Submission history

From: Yujin Jo [view email]
[v1] Tue, 20 Jan 2026 08:04:18 UTC (1,624 KB)
[v2] Sun, 19 Apr 2026 15:15:41 UTC (1,606 KB)
[v3] Fri, 23 Oct 2026 16:39:00 UTC (1,606 KB)