跳到正文
arXiv cs.AI· Chang Sun, Yijun Tian, Zhenwei Tang, Yuchen Deng, Yifan Simon Liu, Chenxu Wang, Wei Wang·· 4 小时前AI 评分29

PACD:面向全模态大模型跨模态幻觉的溯源感知共识解码

PACD: Provenance-Aware Consensus Decoding for Mitigating Cross-Modal Hallucinations in Omnimodal Large Language Models

AI 导读

研究者提出免训练框架 PACD,通过溯源感知共识解码缓解全模态大模型的跨模态幻觉。该方法对分支未达成共识的情况用相关性引导候选仲裁,对达成共识的情况用基于溯源的共识恢复,在四个 Omni-LLM 和五个基准上验证,跨模态幻觉缓解相对标准解码最高提升 27.6%,通用音视频推理平均相对提升 7.7%。

正文 · 原文

View PDF HTML (experimental)

Abstract:Omnimodal large language models (Omni-LLMs) integrate audio, video, and text for multimodal reasoning, yet remain vulnerable to cross-modal hallucinations, where one modality improperly influences predictions about another. Existing training-free decoding methods mitigate these errors through modality perturbation or relevance-based weighting. However, due to interference from irrelevant modality cues during fusion and difficulty detecting shared errors, these methods may suppress correct unimodal predictions and allow erroneous consensus to persist. To address these limitations, we propose Provenance-Aware Consensus Decoding (PACD), a training-free framework for mitigating cross-modal hallucinations. PACD comprises two complementary paths based on whether the branches reach consensus. For non-consensus cases, Relevance-Guided Candidate Arbitration identifies task-relevant unimodal evidence and selectively arbitrates between predictions using alternative candidates. For consensus cases, Provenance-Based Consensus Recovery leverages the outranking principle to evaluate the consistency and strength of layer-wise modality evidence for alternative answers, selectively resolving erroneous consensus under confidence constraints. Extensive experiments on four Omni-LLMs and five benchmarks across two tasks demonstrate the superiority of PACD. Compared to existing methods, PACD achieves up to 27.6% relative improvement over standard decoding across CMM, AVHBench, and SVHalluc for cross-modal hallucination mitigation, and an average relative improvement of 7.7% over best baseline across OmniVideoBench and FutureOmni for general audio-visual reasoning.
Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Cite as: arXiv:2611.22456 [cs.AI]
  (or arXiv:2611.22456v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.22456

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Chang Sun [view email]
[v1] Tue, 17 Nov 2026 09:20:20 UTC (13,256 KB)

来源:arXiv cs.AI · arxiv.org