跳到正文
arXiv cs.CL· Chengpeng Fu, Xiaocheng Feng, Wenshuai Huo, Yichong Huang, Baohang Li, Duyu Tang, Ting Liu·· 2 天前AI 评分26

VEPO:用视觉证据引导的偏好优化缓解 LVLM 多语言幻觉

VEPO: Mitigating Multilingual Hallucinations in LVLMs via Visual Evidence-guided Preference Optimization

AI 导读

VEPO 是一种训练时方法,通过将英语中可靠的视觉 grounding 迁移到其他语言,缓解 LVLM 在低资源语言下的物体幻觉,且无需外部模型。它先用英语查询经中间层跨模态注意力定位关键视觉证据,再构造答案相同但证据充分性不同的偏好对(完整图像 vs 关键区域遮挡图像)做直接偏好优化。在中文、法语、印地语的多语言 POPE 和 MME 上均降低幻觉并提升感知,最低资源语言提升最大。

正文 · 原文

View PDF HTML (experimental)

Abstract:Large vision-language models (LVLMs) achieve strong multimodal performance, yet they remain prone to object hallucination---generating answers that are linguistically fluent but visually ungrounded. This issue is especially severe in low-resource languages, where weaker cross-modal alignment causes models to rely more on language priors than on image evidence. Existing remedies either require costly multilingual annotations or external translation/reward models, or intervene at inference time in ways that can distort the output language. We propose VEPO(Visual Evidence-guided Preference Optimization), a training-time method that transfers reliable visual grounding from English to other languages without external models. VEPO first localizes key visual evidence using English queries via mid-layer cross-modal attention, then constructs preference pairs that share the same correct answer but differ in evidence sufficiency (full image vs.\ key-region masked image). Direct preference optimization on these pairs encourages low-resource inputs to prefer answers supported by complete visual evidence. Experiments on multilingual POPE and MME with Chinese, French, and Hindi show consistent hallucination reduction and perception gains, with the largest improvements on the lowest-resource language. Analyses further confirm that VEPO sharpens question-relevant attention and that evidence-guided masking outperforms random masking.
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2611.18096 [cs.CL]
  (or arXiv:2611.18096v1 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2611.18096

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Chengpeng Fu [view email]
[v1] Fri, 13 Nov 2026 12:23:14 UTC (5,385 KB)

来源:arXiv cs.CL · arxiv.org